You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2009/09/15 06:00:57 UTC
[jira] Created: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
-----------------------------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-979
URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker, tasktracker
Affects Versions: 0.20.1
Reporter: Arun C Murthy
Fix For: 0.20.2
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-979:
---------------------------------------
Release Note: Added support to fallback to new task memory configuration when deprecated memory configuration values are set to disabled. (was: Support the fallback to newer task memory configuration keys when older memory configuration keys are set to default disabled.)
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-979:
------------------------------------
Priority: Blocker (was: Major)
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Attachment: mapreduce-979-20-2.patch
Attached patch for 20 branch.
The patch applies directly on 21.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763337#action_12763337 ]
Sreekanth Ramakrishnan commented on MAPREDUCE-979:
--------------------------------------------------
The changes look fine to me.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Attachment: mapreduce-979-3.patch
Fixing a small javadoc change, changing {{org.apache.hadoop.mapred.JobConf.getMemoryForReduceTask()}} to read correctly.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Release Note: Support the fallback to newer task memory configuration keys when older memory configuration keys are set to default disabled.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Attachment: mapreduce-979-1.patch
Attaching patch for trunk.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764624#action_12764624 ]
Hemanth Yamijala commented on MAPREDUCE-979:
--------------------------------------------
Verified the diff in the 20 patch. +1. The test failure is tracked in MAPREDUCE-1029 and unrelated to this patch. I will commit this.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Attachment: mapreduce-979-1-20.patch
Attaching patch fixing this issue:
if the old key is set to disabled memory limit we fall back to new value.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764569#action_12764569 ]
Hemanth Yamijala commented on MAPREDUCE-979:
--------------------------------------------
+1 for the changes.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764584#action_12764584 ]
Hemanth Yamijala commented on MAPREDUCE-979:
--------------------------------------------
+1 for the 20 patch, other than the correction in the javadoc similar to what's been applied for trunk. Please upload a corresponding 20 patch once the trunk patch passes Hudson.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763369#action_12763369 ]
Hadoop QA commented on MAPREDUCE-979:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421417/mapreduce-979-2.patch
against trunk revision 819740.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/66/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/66/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/66/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/66/console
This message is automatically generated.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755342#action_12755342 ]
Hemanth Yamijala commented on MAPREDUCE-979:
--------------------------------------------
To be clear, mapred-default in hadoop is updated to not have mapred.task.maxvmem. However, M/R applications like PIG could bundle their own mapred-default which, if not updated, would break.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765637#action_12765637 ]
Hudson commented on MAPREDUCE-979:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk #112 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/112/])
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-979:
---------------------------------------
Resolution: Fixed
Assignee: Sreekanth Ramakrishnan
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this to trunk, branch 0.21 and branch 0.20. Thanks, Sreekanth !
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764663#action_12764663 ]
Hudson commented on MAPREDUCE-979:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #72 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/72/])
. Fixed JobConf APIs related to memory parameters to return values of new configuration variables when deprecated variables are disabled. Contributed by Sreekanth Ramakrishnan.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Assignee: Sreekanth Ramakrishnan
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Status: Open (was: Patch Available)
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Status: Patch Available (was: Open)
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Attachment: mapreduce-979-20-3.patch
Attaching branch 20 version of the patch.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-20-3.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764608#action_12764608 ]
Hadoop QA commented on MAPREDUCE-979:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421841/mapreduce-979-3.patch
against trunk revision 824238.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/159/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/159/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/159/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/159/console
This message is automatically generated.
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch, mapreduce-979-20-2.patch, mapreduce-979-3.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated MAPREDUCE-979:
---------------------------------------
Attachment: mapreduce-979-2.patch
The attached patch (mapreduce-979-2.patch) has some modifications I did as part of the review. These include:
- Refactoring the code in getMemoryForMapTask, getMemoryForReduceTask and getDeprecatedMemoryValue. Specifically the last one, I made it return only the deprecated value, or disabled value if deprecated keys are not defined.
- Fixed a bug in getDeprecatedMemoryValue to handle normalization of the config values.
- Modified getMaxVirtualMemoryForTask to behave similarly as the new APIs. So, they return the new values if the deprecated values are undefined.
- Updated Javadocs for the public APIs
- Added unit tests for the negative values in configuration.
Sreekanth, can you please check if these changes are fine ?
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-979)
JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs
when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated MAPREDUCE-979:
---------------------------------------------
Status: Patch Available (was: Open)
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.1
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapreduce-979-1-20.patch, mapreduce-979-1.patch, mapreduce-979-2.patch
>
>
> JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1, this results in failed job-submissions when mapred-default.xml has the default value of -1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.