You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2012/01/20 00:40:40 UTC
[jira] [Created] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Default RPC handlers are very low for YARN servers
--------------------------------------------------
Key: MAPREDUCE-3699
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Fix For: 0.23.1
Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Attachment: MR-3699.1.patch
Trivial patch based on findings from performance benchmarks run on MRv2.
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Status: Patch Available (was: Open)
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194209#comment-13194209 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #429 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/429/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
svn merge --ignore-ancestry -c 1236396 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236398
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194212#comment-13194212 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1597 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1597/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236396
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3699:
-----------------------------------------------
Resolution: Fixed
Release Note: Increased RPC handlers for all YARN servers to reasonable values for working at scale.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and merged it into 0.23. Thanks Hitesh!
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194195#comment-13194195 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3699:
----------------------------------------------------
+1. This looks good.
bq. -1 tests included. The patch doesn't appear to include any new or modified tests.
This is just a configuration change.
I am pushing this in.
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194183#comment-13194183 ]
Hadoop QA commented on MAPREDUCE-3699:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512027/MR-3699.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1689//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1689//console
This message is automatically generated.
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Attachment: MR-3699.2.patch
modified yarn-default.xml and added missing entry to mapred-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Status: Patch Available (was: Open)
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194644#comment-13194644 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #938 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/938/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236396
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194688#comment-13194688 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #173 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/173/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
svn merge --ignore-ancestry -c 1236396 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236398
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194105#comment-13194105 ]
Siddharth Seth commented on MAPREDUCE-3699:
-------------------------------------------
Patch looks good. Hitesh, could you please update the default-values in yarn-default.xml as well.
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Arun C Murthy (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-3699:
----------------------------------------
Assignee: Hitesh Shah
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194678#comment-13194678 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #971 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/971/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236396
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Status: Open (was: Patch Available)
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193674#comment-13193674 ]
Hadoop QA commented on MAPREDUCE-3699:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511954/MR-3699.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1681//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1681//console
This message is automatically generated.
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194210#comment-13194210 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #420 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/420/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
svn merge --ignore-ancestry -c 1236396 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236398
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194218#comment-13194218 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1613 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1613/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236396
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3699) Default RPC handlers are very low
for YARN servers
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3699:
-----------------------------------
Attachment: MR-3699.3.patch
Also, add default val for MR-AM heartbeat to RM into mapred-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194208#comment-13194208 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1669 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1669/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236396
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194653#comment-13194653 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #151 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/151/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
svn merge --ignore-ancestry -c 1236396 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236398
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3699) Default RPC handlers are very
low for YARN servers
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194219#comment-13194219 ]
Hudson commented on MAPREDUCE-3699:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #445 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/445/])
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable values for working at scale. Contributed by Hitesh Shah.
svn merge --ignore-ancestry -c 1236396 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1236398
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
> Default RPC handlers are very low for YARN servers
> --------------------------------------------------
>
> Key: MAPREDUCE-3699
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3699
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3699.1.patch, MR-3699.2.patch, MR-3699.3.patch
>
>
> Mainly NM has a default of 5, RM has 10 and AM also has 10 irrespective of num-slots, num-nodes and num-tasks respectively. Though ideally we want to scale according to slots/nodes/tasks, for now increasing the defaults should be enough.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira