You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/28 02:09:32 UTC
[jira] [Created] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Make use of fadvise in the NM's shuffle handler
-----------------------------------------------
Key: MAPREDUCE-3289
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, nodemanager, performance
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Status: Patch Available (was: Open)
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427052#comment-13427052 ]
Arun C Murthy commented on MAPREDUCE-3289:
------------------------------------------
+1 for the trunk patch, looks good - thanks Todd & Sid.
And +1 for Sid's comment that we should keep trunk and branch-1 in sync.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Attachment: MR3289_trunk.txt
Patch for trunk. Needs some more work for the recent ssl changes. Also needs some testing.
I'm not sure if this is the best way to do this with Netty. Would like some feedback.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427044#comment-13427044 ]
Hadoop QA commented on MAPREDUCE-3289:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538856/MAPREDUCE-3289.branch-1.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2698//console
This message is automatically generated.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237715#comment-13237715 ]
Todd Lipcon commented on MAPREDUCE-3289:
----------------------------------------
Found an issue with this optimization today in the case where some map output partitions are really large (eg on the order of many GBs). Because the fadvise is a blocking call, this can block on the order of minutes before sending the first bytes of response data. This causes the reducer to time out, etc.
Instead, we should fadvise only a "chunk" ahead of the current read position.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3289:
-------------------------------------
Status: Open (was: Patch Available)
I'm not super convinced we want to play God with the buffer-cache, but it's too late to sermonize now...
Minor nits for 23.0:
# Please make the default to false
# Move the config key/default-value to YarnConfiguration, we don't spread them around now.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated MAPREDUCE-3289:
----------------------------------
Attachment: MAPREDUCE-3289.branch-1.patch
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428052#comment-13428052 ]
Hudson commented on MAPREDUCE-3289:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1124 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1124/])
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. (Contributed by Todd Lipcon and Siddharth Seth) (Revision 1368718)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368718
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedChunkedFile.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3289:
-------------------------------------
Fix Version/s: (was: 1.2.0)
1.1.0
I just merged it to branch-1.1 too.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.1.0, 2.0.2-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk_2.txt, MR3289_trunk_3.txt, MR3289_trunk.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415658#comment-13415658 ]
Brandon Li commented on MAPREDUCE-3289:
---------------------------------------
Hi Todd, if you have a new patch, could you please upload it?
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
1.2.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2 and branch-1. Thanks Todd and Brandon!
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148814#comment-13148814 ]
Todd Lipcon commented on MAPREDUCE-3289:
----------------------------------------
Arun, any thoughts on this? Would like to get this in for 0.23.1
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428100#comment-13428100 ]
Hudson commented on MAPREDUCE-3289:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1156 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1156/])
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. (Contributed by Todd Lipcon and Siddharth Seth) (Revision 1368718)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368718
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedChunkedFile.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140478#comment-13140478 ]
Todd Lipcon commented on MAPREDUCE-3289:
----------------------------------------
bq. Move the config key/default-value to YarnConfiguration, we don't spread them around now
On further thought, YarnConfig doesn't seem like the right place for it -- this is an MR config rather than YARN, given it's part of the mapreduce-client-shuffle project. We also configure mapreduce.shuffle.port here. Is MRConfig the right spot for both of these, maybe?
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139552#comment-13139552 ]
Arun C Murthy commented on MAPREDUCE-3289:
------------------------------------------
Todd, have you done any benchmarks with this?
I'm a bit skeptical... one idea might be to keep the whole map-output in the buffer cache (particularly if it's freed up since the Datanode isn't using as much of it) to speed up all of the shuffle for the whole job. Also, you might want to start pre-fetching outputs of other maps of the same job (application) as an optimization.
Thoughts?
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3289:
-----------------------------------
Status: Patch Available (was: Open)
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139640#comment-13139640 ]
Todd Lipcon commented on MAPREDUCE-3289:
----------------------------------------
I've tested the MR1 equivalent of this and it makes a significant difference. I agree we could prefetch other map outputs as well, and generally do a smarter job of ordering the fetches, but this small patch made a good difference on MR1 - so I don't have any reason to believe it wouldn't help in MR2.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427705#comment-13427705 ]
Hudson commented on MAPREDUCE-3289:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2570 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2570/])
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. (Contributed by Todd Lipcon and Siddharth Seth) (Revision 1368718)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368718
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedChunkedFile.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Status: Patch Available (was: Open)
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427726#comment-13427726 ]
Hudson commented on MAPREDUCE-3289:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2617 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2617/])
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. (Contributed by Todd Lipcon and Siddharth Seth) (Revision 1368718)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368718
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedChunkedFile.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421837#comment-13421837 ]
Brandon Li commented on MAPREDUCE-3289:
---------------------------------------
Uploaded the branch-1 patch which combines Todd's two patches and is re-based with the head of branch-1.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Attachment: MR3289_trunk_2.txt
Updated patch fro trunk with some fixes, and fadvise when using ssl.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Status: Open (was: Patch Available)
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423685#comment-13423685 ]
Siddharth Seth commented on MAPREDUCE-3289:
-------------------------------------------
The branch-1 patch looks good. Minor nit - the parameters are named differently between branch-1 and trunk (mapred.tasktracker.shuffle.fadvise, mapreduce.shuffle.manage.os.cache). Since this will likely go into both at the same time, we could just use the trunk parameter names. How's this been done for other such cases ?
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427041#comment-13427041 ]
Brandon Li commented on MAPREDUCE-3289:
---------------------------------------
Updated the branch-1 patch to use the same parameter names as trunk patch.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3289:
--------------------------------------
Attachment: MR3289_trunk_3.txt
Fixes the findbugs warnings.
I've left the configs in ShuffleHandler itself. MRConfig doesn't seem like the right place - since Shuffle is an Aux Service for yarn, which doesn't necessarily need MR classes to run.
Regarding the parameter names - I think it's better to use the same in both patches, as against introducing a parameter in branch-1 and deprecating it immediately in the the branch-2 patch.
Brandon, could you please make this change in the branch-1 patch you had posted.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462872#comment-13462872 ]
Arun C Murthy commented on MAPREDUCE-3289:
------------------------------------------
Matt - if you don't mind, I'd like to merge this into branch-1.1 since it's been well baked-in. Thoughts?
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.0.2-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk_2.txt, MR3289_trunk_3.txt, MR3289_trunk.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-3289.
------------------------------------
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.1.0, 2.0.2-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk_2.txt, MR3289_trunk_3.txt, MR3289_trunk.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3289:
-----------------------------------
Attachment: 3289-2.txt
3289-1.txt
Attaching two patches which apply to some version of branch-1, though probably not the tip of it. These two, together, implement the feature in MR1 with the bug fix mentioned above.
I haven't had time to forward-port the patch to MR2 again with this fix, but it should be reasonably straightforward.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415671#comment-13415671 ]
Todd Lipcon commented on MAPREDUCE-3289:
----------------------------------------
BTW, feel free to steal this from me if you are planning to work on it. I won't have time in the near future.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3289:
-----------------------------------
Attachment: mr-3289.txt
Here's a rather simple implementation that fadvises with WILLNEED just before sending the output, and DONTNEED after sending the output.
We could make it fancier by fadvising "one map output ahead", since in MR2 a single request can fetch multiple outputs, but I'd prefer to leave that for a later JIRA.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated MAPREDUCE-3289:
----------------------------------
Attachment: MAPREDUCE-3289.branch-1.patch
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3289:
-------------------------------------------
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427724#comment-13427724 ]
Hudson commented on MAPREDUCE-3289:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2552 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2552/])
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. (Contributed by Todd Lipcon and Siddharth Seth) (Revision 1368718)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368718
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedChunkedFile.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 1.2.0, 2.2.0-alpha
>
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427033#comment-13427033 ]
Hadoop QA commented on MAPREDUCE-3289:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538853/MR3289_trunk_3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2697//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2697//console
This message is automatically generated.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's
shuffle handler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426910#comment-13426910 ]
Hadoop QA commented on MAPREDUCE-3289:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538835/MR3289_trunk_2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//console
This message is automatically generated.
> Make use of fadvise in the NM's shuffle handler
> -----------------------------------------------
>
> Key: MAPREDUCE-3289
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager, performance
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt
>
>
> Using the new NativeIO fadvise functions, we can make the NodeManager prefetch map output before it's send over the socket, and drop it out of the fs cache once it's been sent (since it's very rare for an output to have to be re-sent). This improves IO efficiency and reduces cache pollution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira