You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Cristina L. Abad (Created) (JIRA)" <ji...@apache.org> on 2011/11/17 21:35:51 UTC
[jira] [Created] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Using mincore to understand the effect of fadvise in the Linux page cache
-------------------------------------------------------------------------
Key: HADOOP-7832
URL: https://issues.apache.org/jira/browse/HADOOP-7832
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Cristina L. Abad
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Cristina L. Abad (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152342#comment-13152342 ]
Cristina L. Abad commented on HADOOP-7832:
------------------------------------------
At this point I just wanted to share it. It can be very useful i evaluating the fadvise effect on the page cache. For example, I ran some simple tests and verified the benefit of using fadvise: for the intermediate files of a simple terasort test (1 slave node, 10G of input data), the fadvise stuff leads to an increase of from 44% to 78% of the average of pages (of intermediate files) already in the page cache. This is an effect of Map input data being flushed out of the page cache periodically. In this case, I am talking about the intermediate files that we ideally want to keep cached as long as possible (before being sent to the reducers) so that the shuffle and sort phase finishes faster. You already showed how this improves job completion time, and this patch shows how/why this happens.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152350#comment-13152350 ]
Todd Lipcon commented on HADOOP-7832:
-------------------------------------
nice analysis - makes sense. Another big benefit that I'm hoping to see from the fadvise stuff is preventing flush-out of dentries and inodes. On clusters with lots of blocks, we see block reports taking many minutes to generate (and incurring lots of seeks) because the FS metadata gets evicted when MR jobs run.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152351#comment-13152351 ]
Todd Lipcon commented on HADOOP-7832:
-------------------------------------
btw, another thing I think would be very interesting to look at: count the percentage of write/flush calls on the DN that take longer than ~10ms. This would serve to separate non-blocking writes (that just dirty pages) from blocking writes (once the max dirty page ratio has been exceeded (/proc/sys/vm/dirty_ratio)). Would be very interested to see if the sync_file_range calls improve this metric. (again it's a theory but not something I've verified)
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156021#comment-13156021 ]
Todd Lipcon commented on HADOOP-7832:
-------------------------------------
Hi Nathan. Yep, that's the workaround we suggested to our users for now - it improves things but doesn't completely solve it.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Cristina L. Abad (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cristina L. Abad updated HADOOP-7832:
-------------------------------------
Description: This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
Priority: Minor (was: Major)
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Cristina L. Abad (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cristina L. Abad updated HADOOP-7832:
-------------------------------------
Attachment: hadoop-7714-20s-205-mincore.txt
Attached patch that can be applied to 20s (205), after applying Todd's 7714 patch. This is experimental and its purpose is to make it available to anyone interested in evaluating the effect (on the page cache) of using 7714. This information (percentage of pages already in page cache) complements performance-based studies (i.e., studying the impact in job completion time).
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155935#comment-13155935 ]
Nathan Roberts commented on HADOOP-7832:
----------------------------------------
bq. On clusters with lots of blocks, we see block reports taking many minutes to generate (and incurring lots of seeks) because the FS metadata gets evicted when MR jobs run.
Todd, have you experimented with /proc/sys/vm/vfs_cache_pressure to deal with this? Seems like datanodes might want to have this value set very low so that dentry/inode caches are significantly stickier than page cache pages. I'm having a hard time coming up with scenarios where this would be detrimental.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152335#comment-13152335 ]
Todd Lipcon commented on HADOOP-7832:
-------------------------------------
Hi Cristina. This is interesting stuff. Is the goal for this to be committed or just posting to share the patch?
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7832) Using mincore to understand the
effect of fadvise in the Linux page cache
Posted by "Cristina L. Abad (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152311#comment-13152311 ]
Cristina L. Abad commented on HADOOP-7832:
------------------------------------------
Sample output (datanode log):
Loaded block blk_4719017558826481280_1001; 86.42578% of its 4K pages were found in OS page cache.
And in tasktracker log (for intermediate files):
Loaded file taskTracker/cabad/jobcache/job_201110251606_0004/attempt_201110251606_0004_m_000000_0/output/file.out; 100.0% of its 4K pages were found in OS page cache.
> Using mincore to understand the effect of fadvise in the Linux page cache
> -------------------------------------------------------------------------
>
> Key: HADOOP-7832
> URL: https://issues.apache.org/jira/browse/HADOOP-7832
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: io, native, performance
> Reporter: Cristina L. Abad
> Priority: Minor
> Attachments: hadoop-7714-20s-205-mincore.txt
>
>
> This Jira adds a way to log the percentage of pages of a file that were on the Linux page cache (when the file is being opened) with the purpose of measuring the effect of issuing posix_fadvise (POSIX_FADV_DONTNEED) calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira