You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2011/11/15 06:29:51 UTC
[jira] [Created] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
ContainerLocalizer should request new resources after completing the current one
--------------------------------------------------------------------------------
Key: MAPREDUCE-3399
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176644#comment-13176644 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #941 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/941/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224970
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176638#comment-13176638 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #142 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/142/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
svn merge -c 1224970 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224971
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176251#comment-13176251 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3399:
----------------------------------------------------
Neat patch, Sid! Looks good to me. +1.
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3399:
--------------------------------------
Status: Open (was: Patch Available)
Canceling - there's some test failures which need to be fixed.
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3399:
-----------------------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: MAPREDUCE-3561
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3399:
--------------------------------------
Attachment: MR3399_v1.txt
Simple patch. Tested on a local cluster - to verify localization is not waiting after a download completes.
Don't think this requires explicit unit tests - functionality is being verified by existing ones.
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176257#comment-13176257 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1542 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1542/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224970
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176256#comment-13176256 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #311 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/311/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
svn merge -c 1224970 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224971
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3399:
--------------------------------------
Attachment: MR3399_v2.txt
Fixes the failing TestContainerLocalizer test.
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176272#comment-13176272 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1491 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1491/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224970
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176273#comment-13176273 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #333 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/333/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
svn merge -c 1224970 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224971
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176258#comment-13176258 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1470 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1470/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224970
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3399:
--------------------------------------
Status: Patch Available (was: Open)
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176261#comment-13176261 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #322 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/322/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
svn merge -c 1224970 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224971
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175194#comment-13175194 ]
Hadoop QA commented on MAPREDUCE-3399:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508449/MR3399_v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1500//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1500//console
This message is automatically generated.
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Arun C Murthy (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-3399:
----------------------------------------
Assignee: Siddharth Seth
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3399:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Release Note: Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and merged into branch-0.23. Thanks Sid!
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176627#comment-13176627 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #121 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/121/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
svn merge -c 1224970 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224971
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3399) ContainerLocalizer should request
new resources after completing the current one
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3399:
--------------------------------------
Status: Patch Available (was: Open)
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3399) ContainerLocalizer should
request new resources after completing the current one
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176631#comment-13176631 ]
Hudson commented on MAPREDUCE-3399:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #908 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/908/])
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224970
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
> ContainerLocalizer should request new resources after completing the current one
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3399
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3399
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3399_v1.txt, MR3399_v2.txt
>
>
> Currently, the ContainerLocalizer to NM heartbeats to the NM every second. Not very significant, but this causes a ~4second delay in jobs (job jar, splits, etc). Instead, it should heartbeat to ask for additional resources to localize as soon as the previous one is localzied. There's already a TODO in the ContainerLocalizer for this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira