You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Peter Schuller (Created) (JIRA)" <ji...@apache.org> on 2012/02/23 01:35:49 UTC
[jira] [Created] (CASSANDRA-3948) SequentialWriter doesn't fsync()
before posix_fadvise()
SequentialWriter doesn't fsync() before posix_fadvise()
-------------------------------------------------------
Key: CASSANDRA-3948
URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
Project: Cassandra
Issue Type: Bug
Reporter: Peter Schuller
This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3948) rename
RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
Posted by "Peter Schuller (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller resolved CASSANDRA-3948.
---------------------------------------
Resolution: Fixed
Committed (1.1 + trunk after all).
> rename RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
> -------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
> Attachments: CASSANDRA-3948-trunk.txt
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3948) SequentialWriter doesn't fsync()
before posix_fadvise()
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3948:
--------------------------------------
Reviewer: scode
Component/s: Core
Fix Version/s: 1.1.0
Assignee: Pavel Yaskevich
> SequentialWriter doesn't fsync() before posix_fadvise()
> -------------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3948) rename
RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219052#comment-13219052 ]
Pavel Yaskevich commented on CASSANDRA-3948:
--------------------------------------------
+1
> rename RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
> -------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
> Attachments: CASSANDRA-3948-trunk.txt
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3948) rename
RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3948:
--------------------------------------
Fix Version/s: (was: 1.1.0)
1.1.1
> rename RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
> -------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.1
>
> Attachments: CASSANDRA-3948-trunk.txt
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3948) SequentialWriter doesn't
fsync() before posix_fadvise()
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216597#comment-13216597 ]
Pavel Yaskevich commented on CASSANDRA-3948:
--------------------------------------------
Exactly, we can't really control (measure) the contents of the page cache so instead we just define intervals for our files when to call fadvice.
> SequentialWriter doesn't fsync() before posix_fadvise()
> -------------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3948) SequentialWriter doesn't fsync()
before posix_fadvise()
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3948:
--------------------------------------
Attachment: CASSANDRA-3948-trunk.txt
Suggesting attached patch to rename the variable to reflect this (trunk only, since no functional change).
> SequentialWriter doesn't fsync() before posix_fadvise()
> -------------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
> Attachments: CASSANDRA-3948-trunk.txt
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3948) rename
RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3948:
--------------------------------------
Summary: rename RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE (was: SequentialWriter doesn't fsync() before posix_fadvise())
> rename RandomAccessReader.MAX_BYTES_IN_PAGE_CACHE
> -------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
> Attachments: CASSANDRA-3948-trunk.txt
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3948) SequentialWriter doesn't
fsync() before posix_fadvise()
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216595#comment-13216595 ]
Peter Schuller commented on CASSANDRA-3948:
-------------------------------------------
So as long as we're only writing few number of very large files, it should in practice work fairly well. Default settings are 30 seconds expiry of dirty buffers, up to 5% of page cache dirty, on Linux.
That said, MAX_BYTES_IN_PAGE_CACHE is thus not really max bytes in page cache, but rather just fadvise interval in bytes.
> SequentialWriter doesn't fsync() before posix_fadvise()
> -------------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3948) SequentialWriter doesn't
fsync() before posix_fadvise()
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214616#comment-13214616 ]
Pavel Yaskevich commented on CASSANDRA-3948:
--------------------------------------------
We algorithm behind it - we do fadvice(fd, <start_position>, 0) after each 128MB of data written, flush is done in the process of each re-buffer (which is each 64KB by default) so we can skip doing sync when we do fadvice() and just use 0 which would hint kernel so skip everything starting from <start_position>.
> SequentialWriter doesn't fsync() before posix_fadvise()
> -------------------------------------------------------
>
> Key: CASSANDRA-3948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3948
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Pavel Yaskevich
> Fix For: 1.1.0
>
>
> This should make the fadvising useless (mostly). See CASSANDRA-1470 for why, including links to kernel source. I have not investigated the history of when this broke or whether it was like from the beginning.
> For the record I have not confirmed this by testing, only by code inspection. I happened to notice it working on other things, so there is some chance that I'm just mis-reading the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira