You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "T Jake Luciani (JIRA)" <ji...@apache.org> on 2010/12/24 19:38:45 UTC
[jira] Created: (CASSANDRA-1902) Migrate cached pages during
compaction
Migrate cached pages during compaction
---------------------------------------
Key: CASSANDRA-1902
URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7.1
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996206#comment-12996206 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. For something like compaction where we are trying to minimize IO it might be not worth it?
Right, because now instead of just saying "this row i just wrote is hot, keep it around" you do all the i/o of writing it, then all the i/o of re-caching the hot rows because you didn't do any of that at write time. Which is basically the same as just letting the cache warm itself up again naturally, except you do it all at once causing even more contention.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012793#comment-13012793 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
bq. With respect to avoiding waiting on GC: the munmap() is still in finalize() we we're still waiting on GC, right? Just not on every possible ByteBuffer (instead only on the MappedFileSegment itself).
This is correct but on other hand we can unmap segment only when we don't need a reader, I can do an explicite method for it but my tests show that unmap in finalize if pretty useful.
bq. BufferedSegmentedFile.tryPreserveFilePageCache() is doing a tryPreserveCacheRegion() for every page considered hot. The first thing to be aware of then is that this will translate into a posix_fadvise() syscall for every page, even when all or almost all pages are in fact in memory. This may be acceptable, but keep in mind that use-cases where all or almost all pages are in cache, are likely to be the ones CPU-bound rather than disk bound.
Documentation for posix_fadvice/madvice calls suggests to do more frequent little requests instead of big requests - kernel in the high possibility going to ignore an advice on the big region.
bq. fadvising entire regions at once instead of once per page might improve that, but I still think the better solution is to just not DONTNEED hot data to begin with (subject to potential limitations to avoid too frequent DONTNEEDs).
We can't stop using DONTNEED while writing compacted file because it will suck pages from sstables which are currently in use. And we do WILLNEED's only when we have SSTableReader for a compacted file ready - right before old sstables going to be replaced with new ones so this is not going to make a big performance impact on the reads. Note that WILLNEED is non-blocking call.
bq. Minor nit: Seemingly truncated doc string for SegmentedFile.complete().
Yes, I will fix that, thanks!
bq. Minor suggestion: Should isRangeInCache() be renamed to wasRangeInCache() to reflect the fact that it does not represent current status? It is not an implementation detail because if it did reflect current reality, the caller would be incorrect (the test on a per-column basis would constantly give false positives as being in cache due to (1) the column just having been serialized, which would be easily fixable, but also because (2) previous columns on the same page, which is more difficult to fix than moving a line of code).
Sounds reasonable to me, I will rename a method.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002617#comment-13002617 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Please rebase with the latest cassandra-0.7 (688ff5bcc0f4c2223eb388ce6199d01403290f5e), can't apply right now.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: (was: CASSANDRA-1902-v4.patch)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030125#comment-13030125 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
Peter,
In the modifications I did for 1902, when we open up SSTable files for compaction, I run mincore across the file to determine which pages are hot already using the hooks in 1902. I use this information in rebuffer() to call DONTNEED on pages that were cold to begin with after a read() call. A percentage of hints given to pages is being ignored by the kernel. Since that page is now 'hot', when we need to mark the hot pages for the new SSTable, we migrate more than we need. For example, we did a test where we made sure that on memtable flushing, we called DONTNEED on entire file. We verified the flushed files were not in cache. Then when compaction kicked in, since all pages were cold, we should have new SSTables that are not in cache. What we observed was, the final file after a large series of flushes + compaction, ended up being 50% in page cache over a long period of time. Even we purposely told the OS we don't want the pages in cache (as we read them).
Jake:
So the problem with that approach is that we still need to make sure as we read data from disk, if the page is cold, it stays cold. Keeping statistics helps the approach of not migrating pages that were cold to hot, but since we still have to read the file during compaction we still need to call DONTNEED on pages that were cold to begin with. That is what is causing the issue, we know a page is cold up front, but the kernel is not respecting that DONTNEED. I thought it might be related to READ AHEAD, so I made sure to fadvise FADV_RANDOM, so that wasn't the issue either.
Jonathan:
Yeah we run with CASSANDRA-2156, that's helped us a lot for performance consistency. We have certain workloads that need to read data recently written, so we disabled calling posix_fadvise(fd, 0, 0) during memtable flushes. We actually found writing new data, and just letting the kernel manage the pages worked better than 1902 solution, because we were calling WILLNEED on pages that were never being read to begin with.
One last thing to try would be keeping track of the last (offset, length) so after read() call fadvise on the previous pair instead of what I just read. The ignoring of hints might be related to the current refcount. I will try this out tonight and update the ticket.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010823#comment-13010823 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
it's created before we know how big it should be so it grows as needed, i have tried both actually.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1902:
--------------------------------------
Reviewer: xedin
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030504#comment-13030504 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Chris,
Looks like I missed at least two things when I looked at the patch last time. Firstly I thought it *was* checking mincore() prior to reading (this is why I said it was not an implementation detail of isRangeInCache() that it does not reflect current reality, since it is called after reading). Secondly, as per the discussion in CASSANDRA-1470 you have fsync/fdatasync prior to DONTNEED in order for the DONTNEED to have any effect. I had a quick look again at the patch and I don't see anything doing this.
Is it possible the effect you're seeing is entirely because of this? IIRC from last time I looked at the source it really looked like it would loop over all pages and invalidate them *AS LONG AS* they aren't dirty.
Note: To be clear, once again just like with direct I/O, there will be a performance penalty associated with the fsync() (approximately the same as the one you see on each write with direct I/O) so benchmarking has to take that into account. The fsync/DONTNEED frequency needs to be balanced with the latency potentially incurred by fsync.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014001#comment-13014001 ]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 3/31/11 3:13 PM:
---------------------------------------------------------------------
What is the difference with the current code?
was (Author: xedin):
What is the different with the current code?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Description:
Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
was:
Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016380#comment-13016380 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Yes, it does read continuously during compaction
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012867#comment-13012867 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
bq. Documentation for posix_fadvice/madvice calls suggests to do more frequent little requests instead of big requests - kernel in the high possibility going to ignore an advice on the big region.
I suspect this comes from WILLNEED being intended as a read-ahead hint. The implementation puts a maximum cap on the amount of read-ahead it'll do:
http://lxr.free-electrons.com/source/mm/fadvise.c#L95
http://lxr.free-electrons.com/source/mm/readahead.c#L203
http://lxr.free-electrons.com/source/mm/readahead.c#L236
My knowledge of the VM subsystem fails me but my interpretation of max_sane_readahead() is that it's trying to essentially cap based on available memory resources such that it may or may not do the read-ahead (or do it to a lesser extent) based on memory pressure.
This is different from e.g. DONTNEED which is fine to call for huge segments and seems to be O(n) (as per CASSANDRA-1470 discussion).
Btw, it's also possible to populate with an mmap() call with MAP_POPULATE (Linux-specific, but we're totally Linux specific anyway). I haven't checked but empirically I suspect this is not subject to limits on how much you populate - and the call is also blocking.
bq. Note that WILLNEED is non-blocking call.
Implementation:
http://lxr.free-electrons.com/source/mm/readahead.c#L145
which boils down to a call to:
http://lxr.free-electrons.com/source/mm/readahead.c#L109
So yes it's non-blocking, but you're still either going to take the random I/O (remember that as of CASSANDRA-1470 we know nothing will be in page cache) or else requests would be dropped somewhere (causing less hotness). If you're taking the I/O, again either the I/O is done in such a way as to essentially be serial or else it's done with some level of parallelism. Which one will affect the impact is has on other I/O.
bq. We can't stop using DONTNEED while writing compacted file because it will suck pages from sstables which are currently in use. And we do WILLNEED's only when we have SSTableReader for a compacted file ready - right before old sstables going to be replaced with new ones so this is not going to make a big performance impact on the reads.
The whole process is going to be suboptimal yes since both the old and the new sstable will be cached. But note that even though the read-ahead is done at the point of the switch, the actual freeing of cached pages associated with the old sstable is still happening at some point in the future as a result of GC (right? or am I misreading?).
My thinking has been that something along the lines of CASSANDRA-1608 and capping sstable sizes may helt mitigate these effects at some point in the future. Meanwhile, the only way I see to avoid double caching is to DONTNEED the old table first, but that has the obvious effects on live traffic.
There is definitely the trade-off here and the timing is different in the WILLNEED-just-before-switch case, maybe leading to more data being retained hot, but at the cost of all that seek-bound I/O (or am I wrong about that bit?) which, for large sstables, will happen over some non-trivial amount of time.
Anyways... there are lots of details and they're difficult to express in a organized fashion. I think the main thing at least is to just be explicitly aware of the seek-bound I/O implied by compaction and probably confirm what the behavior actually is for a large sstable - specifically whether it will cause a storm of highly concurrent I/O with an insanely high queue depth (= *very* bad for live reads) or a slower mostly serial background read-in (= much better for live reads) or somewhere in between.
(I'll try to look into that, subject to time.)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002637#comment-13002637 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Rebased patch does not include org.apache.cassandra.utils.PageCacheMetrics class.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016379#comment-13016379 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Are you reading continuously during compaction? If not you should put that step into a loop.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1902:
--------------------------------------
Fix Version/s: (was: 0.7.4)
0.8
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996145#comment-12996145 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
I'd like to propose a change that might work much better for longer compactions: We write out a new SSTable and using posix_fadvice (like we do today), and make sure nothing is in cache. After the file is done being written, we call getCachedPages() across all sstables used in compaction and compute which pages are hot AFTER compaction is complete. This would allow us to to then sweep through the new SSTable written and mark pages that were hot. If we do the process while file is being written and we have a compaction that might take an hour, by the time it's done, the cache could churn.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: (was: 1902_v1.txt)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012517#comment-13012517 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
Okay, so we *are* adding a copy by using getByteArray. I guess that takes care of the race conditions but it baffles me that copy + wrap can be as fast as duplicate + set position. (duplicate != copy.)
All the order(nativeOrder) call does is set byte order for things like the get* methods, which all look like this:
{code}
private long getLong(long a) {
if (unaligned) {
long x = unsafe.getLong(a);
return (nativeByteOrder ? x : Bits.swap(x));
}
return Bits.getLong(a, bigEndian);
}
{code}
And here is where unaligned gets set:
{code}
static boolean unaligned() {
if (unalignedKnown)
return unaligned;
PrivilegedAction pa
= new sun.security.action.GetPropertyAction("os.arch");
String arch = (String)AccessController.doPrivileged(pa);
unaligned = arch.equals("i386") || arch.equals("x86") || arch.equals("x86_64")
|| arch.equals("amd64") || arch.equals("ppc"); // Mac OS X / PPC: see Radar #3253257
unalignedKnown = true;
return unaligned;
}
{code}
... so byte ordering is basically a no-op on any architecture we are likely to run on.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996199#comment-12996199 ]
Chris Goffinet edited comment on CASSANDRA-1902 at 2/18/11 1:52 AM:
--------------------------------------------------------------------
Couldn't we build a list when reading/writing of the SSTables the first time, so that after new SSTable write is finished, we just mmap through the page cache, and we already know the new locations on the first pass?
was (Author: lenn0x):
Couldn't we build a list when reading/writing of the mappings, so that after write is finished, we just mmap through the page cache, and we already know the new locations on the first pass?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030505#comment-13030505 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
(Without the fsync() I expect the results of DONTNEED to vary on a per-call basis depending on how the kernel happened to flush data to disk. This should vary a lot depending on the CPU vs. disk boundness of the compaction, the DONTNEED frequency in terms of bytes, and the overall I/O load of the system. As such it's difficult to make generalizations about the results, but it's clear that if the intent of page cache hinting is to work as expected in ways you can reason with, the hint needs to be done such that they consistently work or at least work/don't work in ways that are understood)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v6.patch
fixed misspelling. Peter can you write a resume of your previous comment for the mere mortals?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1902:
----------------------------------------
Fix Version/s: (was: 0.7.2)
0.7.3
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v9-trunk-rebased.patch
rebased with latest trunk (eed330ec086f36b36ab7ac2fc74693db43ae86bf)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996166#comment-12996166 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
If we compact several TB, how big can the cachedPages list grow? Do we need to change that to an iterator-based approach to avoid memory pressure from allocating a huge List?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012783#comment-13012783 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Catching up with ticket history and the latest version of the patch, a few things based on the history+patch themselves (I have not tested or benchmarked anything):
With respect to avoiding waiting on GC: the munmap() is still in finalize() we we're still waiting on GC, right? Just not on every possible ByteBuffer (instead only on the MappedFileSegment itself).
BufferedSegmentedFile.tryPreserveFilePageCache() is doing a tryPreserveCacheRegion() for every page considered hot. The first thing to be aware of then is that this will translate into a posix_fadvise() syscall for every page, even when all or almost all pages are in fact in memory. This may be acceptable, but keep in mind that use-cases where all or almost all pages are in cache, are likely to be the ones CPU-bound rather than disk bound.
The bigger issue with the same thing, is that in the cache of large column families that we're trying to optimize for, unless I am missing something the preservation process is expected to be entirely seek bound for sparsely hot sstables. In the best case for mostly-hot sstables it might not be seek bound provided that pre-fetching and/or read-ahead and/or linear access detection is working well, but that seems very dependent on system details and the type of load the system is under (probably less likely to work well under high "live" read i/o loads). In the non-best case (sparsely hot), it should most definitely be entirely seek bound.
fadvising entire regions at once instead of once per page might improve that, but I still think the better solution is to just not DONTNEED hot data to begin with (subject to potential limitations to avoid too frequent DONTNEEDs).
Note: The original motivation for avoiding frequent DONTNEED was performance in relation to the syscall. But in this case we're taking a "one syscall per page" hit anyway with the WILLNEED:s. In fact in the case of a very hot sstable (where CPU efficiency is more important than a cold sstable where disk I/O is more important) the WILLNEED:s should be more numerous than the DONTNEED:s would have been had they been "fragmented" according to a hotness map.
Disregarding the CPU efficiency concerns though, the primary concern I'd have is the WILLNEED calls. Again I haven't tested to make sure I'm not mis-reading it, but this should mean that all compactions of actively used sstables will end, after the streaming I/O, with lots of seek bound reads to fullfil the WILLNEED:s. This can take a lot of time and be expensive in terms of the amount of "disk time" being spent (relative to a rate limited compaction process), and also violates the otherwise preserved rule that "the only seek-bound I/O is live reads; all other I/O is sequential".
Also: If WILLNEED blocks until it's been read, the impact on live traffic should be limited but on the other hand latency should be high under read load. If WILLNEED doesn't block throughput should have a chance of being reasonable by maintaining some queue depth, but on the other hand would potentially severely affect live reads. (I don't know which is true, I should check, but I haven't yet.)
Minor nit: Seemingly truncated doc string for SegmentedFile.complete().
Minor suggestion: Should isRangeInCache() be renamed to wasRangeInCache() to reflect the fact that it does not represent current status? It is not an implementation detail because if it did reflect current reality, the caller would be incorrect (the test on a per-column basis would constantly give false positives as being in cache due to (1) the column just having been serialized, which would be easily fixable, but also because (2) previous columns on the same page, which is more difficult to fix than moving a line of code).
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v9-trunk.patch
branch: trunk, last commit is bc087008c1cd869e50a79febdb11351edfe48890
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996177#comment-12996177 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
... actually the right limit is system memory, since we don't care how much is UNcached.
I'm estimating an upper bound of about 3GB -- say 128GB for a ludicrously large cache = 32M entries, times 96B for each Pair.
On the one hand if you have that much memory you can afford to use an extra 3GB of heap during compaction, but memory pressure that occurs only infrequently can be an unpleasant surprise as we saw for CASSANDRA-2175.
If there's a sane way to do this w/o materializing the entire list let's do that.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996203#comment-12996203 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
But if this is a huge SSTable we need to store all the offsets for all the rows since we don't know which are in the page cache. Now we are talking potentially a lot of memory.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996170#comment-12996170 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
That's worst case too. I don't know of anyone running a node with 1TB+ of memory. Since they would just be hot pages.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902-formatted.txt
OCD alert: attached fixes the style violations I noticed. Also removes catch statements from CLibrary methods that shouldn't be necessary.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v5.patch
Added test for NativeMappedSegment with minor changes in NativeMappedSegment. This can be counted as final
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902-per-column-migration-rebase.txt
rebased
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013996#comment-13013996 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Here's my test results:
1. Loaded in just enough to fit in page cache + flushed
2. Primed reads
3. Overwrite with same data + flushed
4. Ran major compaction while reading as much as possible the entire time.
Read results for 100k during major compaction (migration ON):
{noformat}
19,13,14,14,82,16,15,67,19,14,47,53,74,15,14,46,32,15,13,65,15,14,67,16,13,59,12,11,11
^ ^ ^ ^ ^ ^ ^ ^ ^
{noformat}
vs with migration OFF
~14-16 X times, then 260, 68, 16....
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013659#comment-13013659 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Regarding drop caches: Right, I don't remember whether the echo will block until eviction is complete or not (in cases where it is slow, it should be CPU bound though and not imply I/O). But I made sure that: (1) the echo terminated, (2) I got iostat running, (3) waited for flurry of I/O to complete that is generated by background operations on any modern machine, that you suddenly see when you do a *complete* buffer cache drop, and (4) saw it idle, and (5) only then Cassandra began the pre-population.
So hopefully that part of the test should be kosher.
Regarding mere mortals ;) Sorry. Is it the iostat stuff which is unclear? I'm looking at (in a monospaced font btw, for alignment...) the avgqu-sz column which indicates the average number of outstanding I/O requests for the sampling duration (1 second in this case). This is effectively the "queue depth".
There are usually two main interesting things about "iostat -k -x 1" (-x being key). One is utilization, which shows the percentage of time there was *any* outstanding I/O request to the device. (But one has to interpret it in context; for example a RAID0 of 10 disks can be at "100%" utilization yet be only 10% saturated.) The other is the average queue size, which is a more direct indication of how any concurrent requests are being serviced.
In the case of the 10 disk RAID0, 100% utilization with an average queue size of 1 would mean roughly 10% saturation of underlying disks. 100% utilization with an average queue size of 5 would mean roughly 50% saturation of underlying disks.
The other relevance of the average queue size is on latency. Disregarding potential relative prioritization going on, if the average number of outstanding requests is, say, 10 - any single request will typically have to wait for 10 other requests to be servied first. (But again that has to be interpreted in context; if you have 10 constitutent disks in a RAID0, that 10 is effectively 1 for latency purposes)
So, when judging the expected effects on the latency (and throughput) of "live reads", it's interesting to look at these values.
In particular, consider the simple case of a single seek-bound serial reader. If the average queue depth is 5, this single reader would probably see a throughput roughly 1/5 of normal (I"m assuming otherwise identical I/O in terms of size of requests). A practical example is something like a "tar -czvf" that is reading a lot of small files (fs meta data etc).
So in that sense, a constant pressure of 5 outstanding requests will cause a very significant slow-down to the serial reader.
On the other hand, if instead of having a serial read you have N number of concurrent readers - you would now rather expect a throughput more like N/(N + 5 - 1) of normal. As the concurrency of the interesting I/O increases, the 5 extra makes less of a difference.
You tend to reach an interesting equilibrium here. Suppose you serve some amount of requests per second normally, and that it gives rise to an average queue depth of 0.5. Now add the constant background pressure of 5 requests. Assuming the reads (that normally gave rise to the 0.5 queue depth) are *independent* (I.e., added latency to one does not prevent the next one from coming in), what tends to happen is that you start accumulating outstanding requests until the number of concurrent requests is high enough that you reach the throughput you had before. Only instead of 0.5 average concurrency, you have something higher than 5. Whatever is required to "drown out" the extra 5 enough.
Even if you are able to reach your desired throughput (requests per second) like this, it significantly adds to the average latency of each read. Not only will each read have to contend with the extra 5 background I/O operations always pending, they also have to compete with other concurrent "live" requests.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012104#comment-13012104 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
I will test this to see how this handles hot reads before/during/after compaction
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010817#comment-13010817 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. BitSet is used in the BRAF to store pages that should be migrated (to avoid memory pressure).
Would a bloom filter be even more memory-efficient?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030079#comment-13030079 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Yeah thanks for the info Chris.
Another thought would be to keep read statistics for the last N minutes vs relying on the current cached pages to identify what's active...
and use them to migrate the rows we know are currently active
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002857#comment-13002857 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
So we're still precomputing the active set, using a bloom filter to save memory. Sounds reasonable. But what part is lazy now that wasn't?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014001#comment-13014001 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
What is the different with the current code?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030508#comment-13030508 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
In general I've been thinking some more lately about this whole issue with caching again. Even with perfectly working direct I/O or advising the kernel in ways that actually work, we'll never get around the problem of needing to maintain a much larger amount of data in memory than is the *actual* live set unless we can effect some kind of incremental transition in the live set being served by live reads. No matter how well the migration works, it'll just tend to be seek bound anyway and the longer it takes to re-populate the more the positive effects are negated anyway since you're still keeping the live sstables in-core while also trying to re-populate the new data.
I once filed CASSANDRA-1658 for similar reasons, but as has been pointed out that's a lot of complexity. As jbellis pointed out at the time, CASSANDRA-1608 could help too with limiting live set sizes.
In particular, I'm leaning more and more towards limiting of sstable sizes in combination with cache advising. If sstable sizes are limited one has a more natural and vaguely reasonable unit of data that is transitioned between sstables at a time, and you can must more easily avoid extremities associated with large compactions (e.g., a smaller repair waiting 1 day for huge compaction work that's already submitted).
That said, maybe a "reasonable" sstable size is still too large to be an appropriate unit from the perspective of cache warmness and compaction/repair, which would lead to the need for something like CASSANDRA-1658 anyway. I can see the incremental transition happening in such a way that the live traffic is re-directed to the new sstable in a moving wave-front matched by a DONTNEED/WILLNEED process such that no significant amount of data would have to be uselessly kept in memory at all.
I dunno, there are no really easy answers. I'm currently battling with a situation where we have exactly this as an issue; the active set is too big to realistically rely on row caching, and as soon as you take the leap to rely on page caching you have repair/compaction issues. Particularly when repair effectively transfers most or all of the data set on other nodes, very significantly increasing the live size of the CF during repair work (something else that I hope would be possible to mitigate with smaller sstables, as repairs could be done more incrementally as an iteration over the key range, letting local compaction happen as data is streamed).
Incidentally the DONTNEED:ing in CASSANDRA-1470, to the extent that it works given lack of fsync, is probably working against us and we'll probably patch to disable the behavior. Taking a sudden cache coldness when traffic is switched to a newly written DONTNEED:ed set of sstables is not pretty; and if DONTNEED is successful you have an issue here even if your entire CF + temporary expansion during repair/compaction fits in memory.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996221#comment-12996221 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
bq. Lazy computation really seems better to me.
I need to think about how possible that is since we are iterating through the rows in compaction the pages are being loaded optimistically so it's going to cause some false positives.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Remaining Estimate: 96h
Original Estimate: 96h
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016330#comment-13016330 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Sure, I'm on it.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004695#comment-13004695 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Removing from trunk too... keeping it open to work/refactor this. Frankly the first attempt would have sufficed. I'll edit the description to make the purpose of this issue more clear.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014128#comment-13014128 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
I have an idea of how to avoid mmap at all at SSTableIdentityIterator by using SSTableReader.dfile which is already mmaped in segments, it should lower i/o pressure during compaction with migration ON.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020651#comment-13020651 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Thanks for you comment Chris, let me clarify one thing - compaction process uses BRAF and reads use SSTableReader.dfile which is by default memory mapped, compaction with DONTNEED has a minimal impact on the hot pages even right now it doesn't take away hot pages from the memory mappings (only can re-assign them to the BRAF when kernel experiences lack of the pages), you can check that by yourself using slightly modified mincore program from http://insights.oetiker.ch/linux/fadvise/. I think that we should reject using BRAF for compaction and move everything to use SSTableReader.dfile (SegmentedFile) plus make sure that we don't make "hot" what we don't need, it also has other benefits that I have listed in CASSANDRA-2459.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020171#comment-13020171 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
CASSANDRA-1470 introduced a way to help not pollute page cache. It also didn't solve it 100% We realized that during compaction, DONTNEED is used when reading pages from existing SSTables. While compaction runs this will effectively make all hot pages in those files, which are still being served be marked to be removed from cache. So we certainly helped not pollute cache, we also effectively removed hot data as well while compaction runs. We need to modify this patch so that while reading pages on existing SSTables we be sure to not remove those hot pages during compaction. I am happy to open another ticket if that makes more sense. But I figured we could just go ahead and make this ticket support that functionality too.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012513#comment-13012513 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Pointer.getByteArray gives us the same performance as zero-reads at the current version, I have tried Pointer.getByteBuffer - it's slower than getByteArray + it needs to be re-ordered to Big Endian.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017738#comment-13017738 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Will add that tomorrow!
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996199#comment-12996199 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
Couldn't we build a list when reading/writing of the mappings, so that after write is finished, we just mmap through the page cache, and we already know the new locations on the first pass?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995960#comment-12995960 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
Instead of re-memmapping things, can we use the one we already have in SSTableReader.dfile (assuming mmap mode is turned on)?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996215#comment-12996215 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. 32 * 4k isn't a large value
It's huge for "object data" CFs (vs "materialized view" CFs).
For comparison, postgresql default block size is 8K and you typically have several if not dozens of rows in a single block.
bq. I guess we could base it on some fraction of the avg row size per CF?
That would be a brute force solution but you come back to fairly large overhead to track it all at once. Lazy computation really seems better to me.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018167#comment-13018167 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
I have been thinking about change in SSTableScanner to use SSTableReader.dfile instead of BRAF which can potentially give us the following benefits:
a). don't need to skip a cache because we will be operating on memory mappings
b). better performance (not copying data between kernel and user buffers as effect gained from using memory mapped segments, avoiding time operating on the kernel mode (+ time for switching context and read-ahead pressure) which BRAF involves)
c). less impact on the live-reads
d). less garbage will be created
e). less file descriptors opened
What do you think? (of course, this should be done in the separate task)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030512#comment-13030512 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
There's been a lot of discussion spread over some tickets, and it's a complex issue that is almost impossible to benchmark without analysis of what is actually happening. For example, DONTNEED ala 1470 without migration such as in this ticket is expected to be an improvement for truly large datasets where the DONTNEED (if it worked, with fsync etc) means that the impact on live traffic on long-running compactions as it's running - but you still take the coldness hit once compaction completes.
On the other hand, the very same optimization is a total regression in performance if you have a data set which is specifically relying on being small enough that sstables fit in RAM (with sufficient margin). In this case, prior to CASSANDRA-1470 your data would perhaps remain in memory at all times, while with 1470 you'd expect to get a much more un-even performance as sudden spikes in coldness affect you.
Benchmarking will show whatever you want it to show depending on what parameters you choose ;)
I still think that the two primary problems that need to be dealt with are (1) the direct impact of background I/O and (2) the indirect effects due to caching (basically what I talk about in the original post to CASSANDRA-1882, a ball which I totally dropped on the floor - sorry). The complexity mostly arises from how other factors, such as data set size, read access pattern, I/O scheduling, RAID vs. normal disks, etc, affect the impact of these two underlying issues. The relevance of the effects will also be very different depending on what the intended behavior and goals are of the cluster; for some use-cases, maybe going down to disk is totally fine because there's very little live reads and it's just important to keep latency reasonable for the few requests that do reach the systems rather than optimize for throughput. For other cases, going down to disk is completely unacceptable if the cluster is to cope with the live read traffic.
I'm thinking maybe a more over-arching view is in order. To come up with an approach which will overall give a well-rounded behavior where the fundamental I/O trade-offs are considered, while surrounding behavior like compaction, repair, sstable sizing, avoiding transient explosions in live set size etc are all also taken into consideration.
As an arbitrary start point, imagine a future version of Cassandra which does the following at the same time:
(1) Support concurrent compaction which is de-coupled from CPU concurrency such that there is no need for a maximum concurrency. Small/big jobs could more or less seamlessly run concurrently.
(2) Limited sstable sizes achieving reasonably sized work units for bulk work.
(3) Incremental "streaming" (not in the current sense) repair where we can do one "reasonably sized unit of work" (2) at a time and not stream further data until prior work has completed - including compaction to reconcile, to avoid sudden jumps in data set size.
(4) Compaction work spread out over time as in the throttling code that has been tested a bit by now (was it Chris that reported it worked well?)
(5) The changes coming into 0.8 to make repair only repair the range of the primary replica, making it easier to reason about and schedule repairs in a cluster, and making each repair operation shorter in nature and thus less of an administrative problem in terms of timing w.r.t. other cluster operations.
(6) Compactions, whenever/in whatever speed they do happen, also taking advantage of the limited work unit/sstable sizes in additional to taking steps (whether it is posix_fadvise/fsync/etc, direct i/o, or whatever) to mitigate or eliminate the impact on cache locality on these operations.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996212#comment-12996212 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
32 * 4k isn't a large value. I guess we could base it on some fraction of the avg row size per CF?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030024#comment-13030024 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
CASSANDRA-2156 is definitely a simpler approach. If not optimal, is that at least adequate?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020862#comment-13020862 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
I'm not sure I follow you. Are you saying DONTNEED doesn't work (other than for non-fsync():ed pages just written)?
What's the "read-ahead pressue" in (b) above btw? If anything one of the potential downsides of mmap():ed should be the increased reliance on read-ahead in order to avoid taking additional seeks. Also, why (c) (less impact on live reads)?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020171#comment-13020171 ]
Chris Goffinet edited comment on CASSANDRA-1902 at 4/15/11 3:19 AM:
--------------------------------------------------------------------
CASSANDRA-1470 introduced a way to help not pollute page cache. It also didn't solve it 100% We realized that during compaction, DONTNEED is used when reading pages from existing SSTables. While compaction runs this will effectively make all hot pages in those files, which are still being served be marked to be removed from cache. So we certainly helped not pollute cache by taking in cold data, we also effectively removed hot data as well while compaction runs. We need to modify this patch so that while reading pages on existing SSTables we be sure to not remove those hot pages during compaction. I am happy to open another ticket if that makes more sense. But I figured we could just go ahead and make this ticket support that functionality too.
was (Author: lenn0x):
CASSANDRA-1470 introduced a way to help not pollute page cache. It also didn't solve it 100% We realized that during compaction, DONTNEED is used when reading pages from existing SSTables. While compaction runs this will effectively make all hot pages in those files, which are still being served be marked to be removed from cache. So we certainly helped not pollute cache, we also effectively removed hot data as well while compaction runs. We need to modify this patch so that while reading pages on existing SSTables we be sure to not remove those hot pages during compaction. I am happy to open another ticket if that makes more sense. But I figured we could just go ahead and make this ticket support that functionality too.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995921#comment-12995921 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
Can you give an overview of what the patch does? Where and how you "track contiguous pages per key," why that is the right solution, what that has to do with MAX_BYTES_IN_PAGE_CACHE, and whatever else I don't know enough to ask for yet. :)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978436#action_12978436 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Okay. JIRA's web interface renders "O ( n )" as an O followed by thumbs-down... How insightful :)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v10-trunk-rebased.patch
Rebased with the latest trunk. After Chris's comment it seems that we should just close it with "Won't Fix" as it makes things worse on the real systems.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013680#comment-13013680 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
I got you point, it confirms one more time that making this an option was a good decision to make. Also making WILLNEED as a single call for range instead of the page seems to be a good idea too so kernel can decide what to do depending on situation (it still has a right to not take advice into account).
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021033#comment-13021033 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
What I want here is to use the same data source for compaction and reads and though use a memory mapping cache instead of adding normal I/O via BRAF. MADV_RANDOM flag could be used for SegmentedFile to avoid read-ahead, seeks on mmaped files aren't as expensive as using normal I/O, also using madvice and information from mincore we can leave page cache almost untouched during compaction.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996168#comment-12996168 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
If we have 1TB of data, with 4KB pages, we computed storing those as integers to about 500MB.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017869#comment-13017869 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Which one of the existing MBean's should handle that? Creating ConfigurationMBean just for this one thing does not seem to be a good idea for me and I don't see any applicable...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015973#comment-13015973 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Ran stress tool -o read against a empty db with migration enabled and got the following:
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079403#comment-13079403 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Jason,
We already bypass the page cache (see CASSANDRA-1470)
This ticket is to migrate the "hot" dataset after compaction of the data
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995947#comment-12995947 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
bq. Can you give an overview of what the patch does?
At a high level this is to lessen the affect of sstable compaction and cleanup on reads, it does this by letting compaction figure out what rows in a SSTable are in the OS page cache and make sure that the compacted row remains in the page cache. Remember post CASSANDRA-1470 we are not putting anything in the page cache during compaction.
bq. Where and how you "track contiguous pages per key," why that is the right solution
This is the right solution because the worst case is the same as the current code today. It can really only help because it's just giving the OS hints, it's upto the OS to do with that info what it thinks is best.
The important piece is in CLibrary.getCachedPages(File file, int minContiguousPages)
This takes a file and mmaps it in 2G chunks then uses the posix mincore() call to get a vector of which pages in the range are actually cached (for a totally unread file this is []). We use the starting offset + (pagecache_size * each mapped page) to return a vector of positions on disk. we use the minContiguousPages to filter down the noise of cache fragments.
Jump to SSTableScanner, here we use the file positions from getCachedPaged to figure out if a given row is considered "active". If it is we set the isInPageCache flag on the SSTableIdentityIterator.
Jump to CompactionManager, if any part of a row has been flagged as active then we make sure when we write the new SSTable this rows data is not forced out of the page cache (the default action from CASSANDRA-1470)
The two variables we probably should expose here are:
1. BRAF.MAX_BYTES_IN_PAGE_CACHE - this says how many bytes should i let the page cache buffer before I force a flush of the OS cache for this files working (this is currently set to 128mb which, based on my testing is a nice default)
2. SSTableScanner's call to getCachedPages uses a minContiguousPages setting of 32. Again this is a nice default I've found.
By increasing (1) you pollute your page cache more but slightly increase your write performance.
By increasing (2) you migrate less and less rows during compaction.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004068#comment-13004068 ]
Hudson commented on CASSANDRA-1902:
-----------------------------------
Integrated in Cassandra-0.7 #357 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/357/])
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010967#comment-13010967 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Note: I should have marked it with "initial", still have some work do to around it but it is already pretty good as a showcase :)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v8.patch
removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 100000
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
---
after compaction
13 12 14 13
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v3.patch
With whitespaces cleaned up and minor code cleanup to match code style.
+1
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016453#comment-13016453 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Ok, reran on my box (4 core 4 gigs of ram). I also compared it to the previously committed version of this patch (v3) I think v3 performs closer to the way we want. it just needs to perserve the page list similar to the way v7 does and migrate them all at the end, just before the new sstables are swapped in.
{noformat}
./bin/stress -n 100000 -c 10 -S 1024 -k
(ON) 226
(OFF) 229
(OLDON) 230
#flush
./bin/stress -n 100000 -o read
(ON) 34, 15
(OFF) 80, 15
(OLDON) 80, 15
./bin/stress -n 100000 -c 10 -S 1024 -k
(ON) 226
(OFF) 228
(OLDON) 226
./bin/stress -n 100000 -o read
(ON) 15
(OFF) 110, 15
(OLDON) 75, 15
#compact
#active reads
(ON) 110, 122, 153, 111, 66, 34, 96, 15
(OFF) 235, 134, 83, 60, 50, 90, 80, 20
(OLDON) 21, 15, 15, 15, 15, 15,…, 119, 40, 15
{noformat}
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996210#comment-12996210 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. we use the minContiguousPages to filter down the noise of cache fragments
Doesn't this make it basically useless for skinny-row sstables, where rows don't span multiple fragments and hot rows are scattered randomly throughout a sstable with many cold rows?
I think we might need a more sophisticated approach to balance this vs the increased memory usage of tracking individual pages.
Coming back to a pseudo-iterator approach: how about SSTableScanner grabbing 256MB or so of mincore info at a time, lazily during the iteration instead of at construction time? (The smaller the unit of mincore computation, the less stale our hotness data will be, which addresses Chris's concern too.) Then we could track even single-page hotness ranges w/o worrying about memory pressure.
Additional note: not a huge fan of exposing the MAX_BYTES_IN_PAGE_CACHE as a tunable, let's find out where diminishing returns set in (128MB apparently?) and hardcode it up. My experience is that super obscure options like this are not useful to users, it's our job to find a good value.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Comment: was deleted
(was: Ok. well It's merged to trunk so we'll keep it there... )
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani reopened CASSANDRA-1902:
---------------------------------------
Ok. well It's merged to trunk so we'll keep it there...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016173#comment-13016173 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
I reran the heavy read test and it still did the same as before.
Can you please try running the test yourself. I need to do more comparisons with the default to see which really is worse...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015815#comment-13015815 ]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/5/11 9:42 AM:
--------------------------------------------------------------------
uses SSTableReader.dfile by mincore instead of re-mmaping regions all the time in the mmaped mode, if mode is standard uses CLibrary.getCachedPages() method
was (Author: xedin):
uses SSTable.dfile by mincore instead of re-mmaping regions all the time in the mmaped mode, if mode is standard uses CLibrary.getCachedPages() method
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978450#action_12978450 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Right, *not* DONTNEEDing the pages needed should prob be fine just though will need might ensure these pages are kept over the presumably active older sstables.
So you are recommending a binary flag like "compaction.hotness.migration.thresh = .5"? I was thinking the same for simplicity sake.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1902:
--------------------------------------
Fix Version/s: 0.7.1
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978435#action_12978435 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Should it not be a matter of *not* DONTNEED:ing the relevant ranges during write, rather than using WILLNEED? WILLNEED seems to result in read-ahead and is presumably meant for cases where you're reading data in that is expected to not be in cache and want to inform the kernel about the desire for read-ahead.
With respect to contiguous ranges: If possible it would be nice if the desired contiguousness would be a configuration option. For workloads where disk I/O is very critical, I would not be surprised if even the worst-case overhead of very frequent posix_fadvise() calls is going to be worth it since the cost of cache misses is so extremely high relative to the cost of a syscall.
It seems (looking again at http://lxr.free-electrons.com/source/mm/fadvise.c#L118) that posix_fadvise() overhead should be the usual syscall overhead plus O(n) on the number of pages.
As to accomplishing advice on contiguous ranges, I'm not sure what the best course of action is. On a sparsely in-core sstable there is no issue. On a very in-core sstable, DONTNEED:s are probably very often going to be so small as to mean that any thresholding just turns it off almost completely.
I suppose there is some middle-ground somewhere where being smart about fadvise() may be advisable.
For simplicity's sake, how about having a simple threshold; "posix_fadvise() in such a way that no call applies to less than N pages of data". The implementation could be as simple as skipping ("skip" meaning, "dont avoid DONTNEED" - double negation...) ranges that do not fulfill the criteria. The expected result I see from an operational perspective is:
* For very hot tables, hotness remains reasonable because in effect, no DONTNEED is done. You're less efficient of course, but no worse than currently.
* For less hot tables, you start seeing an effect, with the effect being largest on large sparsely cached tables (which is also where it is most important).
* If you are in a position where you really want to squeeze that last bit of hotness out of compaction even if it costs lots of syscalls, you can set the threshold to 0.
I am operating on the assumption that the only motivation for preferring contiguous ranges is the performance penalty of the fadvise().
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v4.patch
This patch makes lazy requests for the page cache info in the SSTableIdentityIterator.
BitSet is used in the BRAF to store pages that should be migrated (to avoid memory pressure).
Actual cache migration happens when SSTableReader gets created for the compacted file - if disk access mode is MMAP each of the segments will get advice using madvice, the same happens for "standard" mode but posix_fadvice is used.
Major change - when we have JNA and disk access mode is MMAP segments will be made using native mmap, this gives us a better control over the mapping (don't need to wait for GC to deallocate mapping, we can force it now using munmap method) this also means that mmap segment will be cleaned automatically so no more long hangs on GC and unused MappedByteBuffer won't hold the cache.
I was testing on the server with 2GB memory (hosted on rackspace): if data fits in the memory it will give you the same performance after compaction as it was before on the other side if data does not fit into memory I've experienced 3-4 times faster reads after compaction.
@tjake: can you test this on your Xen server?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000516#comment-13000516 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Attached new version based on previous feedback.
This takes a lazy approach and migrates pages per column. So it will handle active column slices etc.
The page cache info of a sstable is now stored in a OpenBitSet so a 1TB SSTable will fit in ~23Mb assuming a 4kb page size.
Here are the stats with a 1470 + 1902 flag off and on:
{code}
#write 100k wide rows to CF1
python stress.py -S 3000 -n 100000 -k
(off) 0.7.3: 81
(on) 0.7.3: 93
#flush the keyspace wait for compaction to finish
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
(off) 0.7.3: 73, 13
(on) 0.7.3: 65, 13
#write 100k wide rows to CF2
python stress.py -S 3000 -n 100000 -k -y super
(off) 0.7.3: 79
(on) 0.7.3: 99
#wait for compaction to finish
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
(off) 0.7.3: 227, 36, 14
(on) 0.7.3: 97, 31, 14
#overwrite 100k wide rows to CF1
python stress.py -S 3000 -n 100000 -k
(off) 0.7.3: 100
(on) 0.7.3: 123
#perform major compaction on CF1
#this will test cache migration
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
(off) 0.7.3: 148
(on) 0.7.3: 97
{code}
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016551#comment-13016551 ]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/6/11 9:37 PM:
--------------------------------------------------------------------
removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 100000 (server 4 GB RAM, Quad-Core AMD Opteron(tm) Processor 2374 HE)
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
--- after compaction ----------
13 12 14 13
was (Author: xedin):
removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 100000
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
---
after compaction
13 12 14 13
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030053#comment-13030053 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
What are you saying the kernel is ignoring - the original DONTNEED on the "source" tables or the WILLNEED on the "destination" tables? I'm having trouble seeing how either results in a cumulative over-migration of pages over time.
Do you btw have any observations with respect to the WILLNEED loop becoming seek bound and the impact on live traffic (balance between WILLNEED induced I/O and regular I/O)?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978459#action_12978459 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Recommend is a strong word, but it was at least my initial reaction for the simplicity of it. A threshold expressed in pages or bytes. I am definitely in favor of an option above a hard-coded threshold, but that's coming from the perspective of expecting to deploy in cache sensitive scenarios where CPU will not be the main issue.
As for WILLNEED: I don't know enough about the kernel's page cache eviction to say whether a write + WILLNEED will affect eviction relative to a write only. Either way is plausible I think, without knowing. (e.g. if both write() and WILLNEED is essentially counted as a 'hit' it might help; on the other hand it might not if only page accesses are counted...). In either case, it's another syscall.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012520#comment-13012520 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
the story behind ordering I mentioned is that Pointer.getByteBuffer always ensures native byte order and MappedByteBuffer always has Big Endian order so we need to ensure byte order of the Pointer.getByteBuffer by setting it to Big Endian, but this is not a case here - on my tests on the server with 2gb RAM hosted on rackspace and on the server with high-memory and medium server hosted on ec2 both gave me a better performance using Pointer.getByteArray and wrap comparing to Pointer.getByteBuffer (and the getByteArray performance is almost identical to our current version).
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079401#comment-13079401 ]
Jason Rutherglen commented on CASSANDRA-1902:
---------------------------------------------
I think we can revive this issue leveraging the [same] work Lucene has performed in this area at [1]. Apparently O_DIRECT is the flag that will work. Perhaps we should open a new issue?
1. https://issues.apache.org/jira/browse/LUCENE-2500
2. https://builds.apache.org/job/Lucene-trunk/javadoc/all/org/apache/lucene/store/DirectIOLinuxDirectory.html
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902-per-column-migration.txt
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: (was: 1902-per-column-migration-rebase.txt)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029983#comment-13029983 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
Is it possible to get a rebase of this? If it's easy (manual patch apply gives rejects but I suspect a git rebase will be trivial or significantly easier to deal with).
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030535#comment-13030535 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
Peter,
Just to be clear, I know about the fsync + dirty buffers issue. My patch made sure to fsync data as it flushes. And this isn't related to what I was seeing, because I was watching the pages of SSTables that were all cold, slowly become hot during compaction.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995963#comment-12995963 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
I tried that first but it wouldn't handle standard reading (which still works now).
The cost isn't high at all, calling mincore is fast. Doesn't actually read anything.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996185#comment-12996185 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
If you look at the implementation of CLibrary.getCachedPages you'll see that it only stores the starting offset + number of contiguous pages > min parameter passed in. So worst case you wont get anything near a int per page cache. We also call mincore on 2G chunks of the file at a time (max).
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v7.patch
uses SSTable.dfile by mincore instead of re-mmaping regions all the time in the mmaped mode, if mode is standard uses CLibrary.getCachedPages() method
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013665#comment-13013665 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
So as an example. Suppose you have live read traffic that results in 0.5 average queue size. Assume for simplicity a single underlying disk that does exactly one request at a time and does no optimization for locality on disk. 0.5 average under those circumstances means you're effectively utilizing 50% of the I/O capacity - the disk is spending 50% of it's wall clock time servicing your reads.
Suppose now that you add a background reader that does nothing but generate seek bound small reads, one after another. That, if left alone, would generate an average queue size of 1.
Running these at once should roughly even out. They should get about 0.5 each.
But suppose you run two background readers. In order for your live reads to "get" 50% of the time the disk spends doing reads, live reads must again have as many outstanding requests on average as the background readers. The background readers have 2 outstanding, so your live traffic must maintain two outstanding requests as well.
This means that at any given moment you need a queue depth of 4, two of which are live traffic. So when a new live request comes in, again absent prioritization/re-ordering, it must now wait for 4 requests to complete before it can be serviced. Thus, request time becomes 5x instead of 1x (x being the time needed to perform a read) because it needs to do the read itself, in addition to 4 other reads.
Now, in reality you have a queue depth that is preserved down onto a SAS/SATA bus, and you have disks that will optimize I/O (completely random seek bound I/O can get about twice as fast in terms of throughput (reqs/second) when queue depth starts to reach 32-64 requests - unless this has changed since a few years ago). With RAID you have multiple constituent drives which changes things. SSD:s complicate things too by being capable of actual real physical concurrent I/O.
So the above example would never reflect reality in terms of concrete numbers, but the "effect" is there - the details are just different.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996195#comment-12996195 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
bq. After the file is done being written, we call getCachedPages() across all sstables used in compaction and compute which pages are hot AFTER compaction is complete. This would allow us to to then sweep through the new SSTable written and mark pages that were hot. If we do the process while file is being written and we have a compaction that might take an hour, by the time it's done, the cache could churn.
I like it in theory, I guess the only thing is it's less efficient since you need to re-iterate through all old sstables twice, once for compaction and once for matching the cached pages to the rows. Then you'd need to iterate through the new sstable to find the new rows location.
For something like compaction where we are trying to minimize IO it might be not worth it?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016370#comment-13016370 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
My results comparing OLD and NEW (with migration enabled) (rackspace server with 2GB ram the same as in prev. comments):
script (stress.java) I run (while last compaction goes I run "-n 100000 -o read" in the separate terminal)
{noformat}
./bin/stress -n 100000 -S 1024 -c 10
sleep 50
./bin/stress -n 100000 -S 1024 -c 10 -o read
./bin/stress -n 100000 -S 1024 -c 10 -o read
./bin/stress -n 100000 -S 1024 -c 10
../../bin/nodetool -h localhost flush Keyspace1 Standard1
echo "start reading"
../../bin/nodetool -h localhost compact Keyspace1 Standard1
{noformat}
Results (OLD | NEW):
{noformat}
191 | 182 (write)
22 | 22 (read)
16 | 16 (read)
184 | 182 (write)
171 | 176 (reads during compaction)
55 | 39 (read after compaction was finished)
{noformat}
I don't post results for new code with migration turned off because it shows the same results as OLD code except last read after compaction shows improvement - 44 vs 55.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978450#action_12978450 ]
T Jake Luciani edited comment on CASSANDRA-1902 at 1/6/11 1:46 PM:
-------------------------------------------------------------------
Right, *not* DONTNEEDing the pages needed should prob be fine just thought WILLNEED might ensure these pages are kept over the presumably active older sstables.
So you are recommending a binary flag like "compaction.hotness.migration.thresh = .5"? I was thinking the same for simplicity sake.
was (Author: tjake):
Right, *not* DONTNEEDing the pages needed should prob be fine just though will need might ensure these pages are kept over the presumably active older sstables.
So you are recommending a binary flag like "compaction.hotness.migration.thresh = .5"? I was thinking the same for simplicity sake.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986205#action_12986205 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Oh, the setup above is on a 4 core Xen box with 4G ram the -S 3000 creates about 1.8G of data which is just enough to fit in the unused memory.
So if you test with a larger box be sure to up this.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 1902_v1.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012496#comment-13012496 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
why do we need normalmappedsegment as well as native? could we get rid of normal?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030547#comment-13030547 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
Just to give an update, I did manage to get the hints to work, as long as I called posix_fadvise on the previous read pair. The next problem unfortunately is that the kernel is ignoring the FADV_RANDOM flag I call when the file is open. So read ahead still applies and has an effec This causes problems, because a) its going to be different on many kernels b) I need to see if its even possible to find this value without searching through /proc). I am still going to be -1 on this patch because it's becoming more and more complex to try to manage the state of the page cache. We have other areas in the code as well (streaming of files) that we would need to support as well, because of the fact we aren't checking pages/and making sure we aren't polluting the cache there. In our production environment, we already see the effects of this.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v10-trunk-rebased.patch, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12977801#action_12977801 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Well the good news is the mincore() stuff works via JNA!
I'm re-considering what todo with this information. It's most efficient to keep around contiguous chunks of pages so the plan might be to find the most densely populated ranges of pages in the sstable then get the range of rows this covered. Then pass to the SSTableWriter this list which will subsequently mark the new data written for these pages as POSIX_FADV_WILLNEED. The ordering of the keys should be close.
I think in the average case this will get the most active data cached. It may keep too much data in the page cache though.
Any thoughts on this?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Assignee: Pavel Yaskevich (was: T Jake Luciani)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v9.patch
v9 adds a last improvement - skip old sstables data file page cache right after they are replaced by compacted file.
Benchmarks comparing (current 0.7/new with migration OFF/new with migration ON), server is the same as previously and benchmark it the same too:
{noformat}
CUR|OFF|ON
201|189|188 (write)
22 |22 |17 (read)
14 |14 |14 (read)
189|189|189 (write)
--- reads during compaction ---
CUR: 81, 30, 18, 17, ..., 59, 28, 14, 34, 40
OFF: 54, 32, 23, 15, ..., 54, 30, 18, 38, 40, 35
ON : 22, 16, 16, 16, ..., 15, 15, 28, 18, 15, 13
--- reads after compaction ---
CUR: 35, 28
OFF: 35, 22
ON : 13, 12
{noformat}
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015973#comment-13015973 ]
T Jake Luciani edited comment on CASSANDRA-1902 at 4/5/11 3:58 PM:
-------------------------------------------------------------------
Minor issue: Ran stress tool -o read against a empty db with migration enabled and got the following:
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
was (Author: tjake):
Ran stress tool -o read against a empty db with migration enabled and got the following:
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
WARN 15:55:31,513 madvice(DONT_NEED) failed, errno (12), length 11.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Remaining Estimate: 32h (was: 96h)
Original Estimate: 32h (was: 96h)
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.1
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002637#comment-13002637 ]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 3/4/11 3:00 PM:
--------------------------------------------------------------------
Rebased patch does not include org.apache.cassandra.utils.PageCacheMetrics and PageCacheInformer classes.
was (Author: xedin):
Rebased patch does not include org.apache.cassandra.utils.PageCacheMetrics class.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002895#comment-13002895 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Well I guess its not lazier but it's more iterative, rather than assume a certain number of pages as a minimum and caching the entire row, it caches per column...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013632#comment-13013632 ]
Peter Schuller commented on CASSANDRA-1902:
-------------------------------------------
(isPageCaheMigrationEnabled is misspelled.)
I tested some on 2.6.32 (it's what I have easily available on running physical h/w). I used stress.py as per the earlier benchmarks in this ticket, but after initial population I just used 'nodetool compact' to trigger compaction. (I tested with standard I/O because I happened to make the changes to the BSF (below), and I also realized I'm not sure where the pre-pop happens for the mmap case. Will have to revisit.)
I added some logging and a sleep to BufferedSegmentedFile.tryPreserveFilePageCache() (I'll attach the patch to that file only, it's only for testing purposes). By doing 'sync ; echo 1 > /proc/sys/vm/drop_caches' I artificially triggered the newly written file not being in page cache at all (essentially simulating compacting a sufficiently large sstable that <10% (or such) fits in ram).
A compaction after letting:
python contrib/py_stress/stress.py -S 3000 -n 100000 -o read
run for a while (~ 5 thousand progress), with the above drop cache / compaction, gave:
WARN 22:06:56,723 about to preserve, please drop caches
WARN 22:07:06,723 begin preserve
WARN 22:07:34,299 end preserve
So firstly, this shows WILLNEED being blocking (I guess I should have looked closer, or it's different due to the .32 kernel).
During this time I observed "iostat -x -k 1". There was an initial flurry of I/O with queue depth > 100, but it then settled for a queue depth of around 5-6 or so for the remainder of the period. Here is a fairly typical second (no pretty graphs I'm afraid):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 6.00 0.00 152.00 0.00 61716.00 0.00 812.05 5.05 33.13 6.16 93.60
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017879#comment-13017879 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
CompactionManager MBean...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902_v1.txt
Attaching patch and benchmark test.
The approach I decided on was to track contiguous pages and track this information per key.
This is working for reads (especially the last step) we can see a significant improvement by migrating cached pages. However this comes at the cost of write performance, since we are increasing the TLB flushes. Reading the kernel mailing lists this posix_fadvise should be non-blocking but it can block sporadically when writing FS metadata.
I think the safest thing is to expose the BRAF.MAX_BYTES_IN_PAGE_CACHE since this will directly affect the read vs write io tradeoff.
I can run some tests with different values so we can get an idea of what affect this has.
Could someone else run the below? (0.7.1+) run includes this patch. I'm not sure if this is limited to my testing env.
{code}
#write 100k wide rows to CF1
python stress.py -S 3000 -n 100000 -k
0.7.0: 73
0.7.1: 92
0.7.1+: 98
#flush the keyspace wait for compaction to finish
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
0.7.0: 98, 16
0.7.1: 104, 14
0.7.1+: 69, 14
#write 100k wide rows to CF2
python stress.py -S 3000 -n 100000 -k -y super
0.7.0: 83
0.7.1: 90
0.7.1+: 98
#wait for compaction to finish
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
0.7.0: 262, 53, 23
0.7.1: 41, 14
0.7.1+: 119, 40, 14
#overwrite 100k wide rows to CF1
python stress.py -S 3000 -n 100000 -k
0.7.0: 92
0.7.1: 104
0.7.1+: 99
#perform major compaction on CF1
#read from CF1 till in page cache
python stress.py -n 100000 -k -o read
0.7.0: 173, 29
0.7.1: 186, 30
0.7.1+: 80, 15
{code}
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 1902_v1.txt
>
> Original Estimate: 32h
> Remaining Estimate: 32h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v9-trunk-with-jmx.patch
2 new methods to query and control page cache migration were added to the CompactionManagerMBean.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v4.patch
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995861#comment-12995861 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
Attached new version with a configuration to enable/disable(default) this and CASSANDRA-1470 if you are going to test you need to set enable_page_cache_migration: true in cassandra.yaml
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016503#comment-13016503 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
Ok, I will continue working on this to get "on compaction" reads as fast as they should be
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich updated CASSANDRA-1902:
---------------------------------------
Attachment: CASSANDRA-1902-v6.patch
Fixed doc string for SegmentedFile.complete() and renamed isRangeInCache() to wasRangeInCache().
I have made changes to NativeMappedSegment.migrateCachedPages() and BufferedSegmentedFile.tryPreserveFilePageCache() so it does not do syscall for each page, instead it collectes continues regions of the cache and does a single call on each to minimise seeking as much as possible.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902-BufferedSegmentedFile-logandsleep.txt
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages
during compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016551#comment-13016551 ]
Pavel Yaskevich edited comment on CASSANDRA-1902 at 4/6/11 9:38 PM:
--------------------------------------------------------------------
removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 100000 (server 4 GB RAM, Quad-Core AMD Opteron(tm) Processor 2374 HE)
{noformat}
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
--- after compaction ----------
13 12 14 13
{noformat}
was (Author: xedin):
removed @finalize from MappedFileDataInput which was dropping page cache. Now I see the following results with migration enabled:
options: -S 1024 -c 10 -n 100000 (server 4 GB RAM, Quad-Core AMD Opteron(tm) Processor 2374 HE)
188 (write)
16 (read)
14 (read)
191 (write)
--- reads during compaction ---
21 16 16 27 41 22 16 15 14 15 16 18 14 13
--- after compaction ----------
13 12 14 13
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012511#comment-13012511 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. NormalMappedSegment is used in cases when we don't have a page size (JNA is not installed or not on Linux)
Makes sense.
bq. using NativeMappedSegment we don't need to duplicate buffers on slice calls
Don't we still need to duplicate, in case we unmap the sstable we read from before we return the data to the requester?
Similarly, if we manually munmap, isn't there a race condition where we say "give me the list of sstables" and then while reading one gets compacted and unmapped?
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030013#comment-13030013 ]
Chris Goffinet commented on CASSANDRA-1902:
-------------------------------------------
We've been testing this patch out for awhile with some modifications. It actually makes things worse. First off, it does not check if a page is hot/cold before reading. In our tests we went ahead and added that support so we can make sure pages stay in a consistent state. What we are seeing though is that the kernel is ignoring a small percent of hints on a set of pages during the compaction process. Because of this, this patch ends up migrating pages that it thinks are hot into the new SSTables over time. You can start to really see the effects when you are running for days and weeks. Unfortunately since the kernel is ignoring hints on some of the pages and we can't really have full control I am -1 on this ticket all together after spending weeks on this. We found just letting the OS deal with page management to be better. We originally thought this was going to be better, and in many synthetic benchmarks it has promise. But after running it in a real production system, the latency is much higher.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk-rebased.patch, CASSANDRA-1902-v9-trunk-with-jmx.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.2
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902_v1.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Pavel Yaskevich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012506#comment-13012506 ]
Pavel Yaskevich commented on CASSANDRA-1902:
--------------------------------------------
NativeMappedSegment if used to give us a better control over memory-mapped region of the file, especially for skipping page cache using (madvice(DONT_NEED) and utilizing a mapping by native way - munmap and doing page cache migration. NormalMappedSegment is used in cases when we don't have a page size (JNA is not installed or not on Linux). I have added those methods because it's not possible to clean page cache from the region of the MBB, also using NativeMappedSegment we don't need to duplicate buffers on slice calls.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017735#comment-13017735 ]
T Jake Luciani commented on CASSANDRA-1902:
-------------------------------------------
One thing that you should do here is make the canMigratePageCache option be configurable at runtime via a MBean. I think this would be useful for folks wanting to test out this option without restarting the cluster...
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, CASSANDRA-1902-v6.patch, CASSANDRA-1902-v7.patch, CASSANDRA-1902-v8.patch, CASSANDRA-1902-v9-trunk.patch, CASSANDRA-1902-v9.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process. This is now important since CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from active CFs. This can be done my monitoring which Old SSTables are in the page cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004568#comment-13004568 ]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
I'm not comfortable w/ the amount of code changes that aren't guarded by enable_page_cache_migration. Reverted from 0.7
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.8
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, CASSANDRA-1902-v3.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1902) Migrate cached pages during
compaction
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated CASSANDRA-1902:
--------------------------------------
Attachment: 1902-per-column-migration-rebase2.txt
oops, added v2
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.4
>
> Attachments: 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, 1902-formatted.txt, 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that uses the posix mincore() function to detect the offsets of pages for this file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os cache and make sure the subsequent pages in the new compacted SSTable are kept in the page cache for these keys. This will minimize the impact of compacting a "hot" SSTable.
> A simpler yet similar approach is described here: http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira