You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2008/02/01 10:41:08 UTC
[jira] Created: (HADOOP-2765) wrapper for streaming tasks
wrapper for streaming tasks
---------------------------
Key: HADOOP-2765
URL: https://issues.apache.org/jira/browse/HADOOP-2765
Project: Hadoop Core
Issue Type: New Feature
Components: contrib/streaming
Affects Versions: 0.15.3
Reporter: Joydeep Sen Sarma
here's the motivation:
we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
Submiting patch that fixes javac, javadoc and findbugs warnings.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577825#action_12577825 ]
Hudson commented on HADOOP-2765:
--------------------------------
Integrated in Hadoop-trunk #426 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/426/])
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575825#action_12575825 ]
Devaraj Das commented on HADOOP-2765:
-------------------------------------
*sigh* This patch doesn't work on cygwin. ulimit seems to be not well supported there. I propose we disable this ulimit thing for hadoop on cygwin.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye updated HADOOP-2765:
-------------------------------------
Fix Version/s: (was: 0.17.0)
0.16.1
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574349#action_12574349 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376951/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 8 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1883/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1883/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1883/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1883/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-2765:
--------------------------------------------
Release Note: This feature enables specifying ulimits for streaming/pipes tasks. Now pipes and streaming tasks have same memory available as the java process which invokes them. Ulimit value will be the same as -Xmx value for java processes provided using mapred.child.java.opts.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571309#action_12571309 ]
Devaraj Das commented on HADOOP-2765:
-------------------------------------
Some comments:
1) I don't see why we should deprecate captureOutAndErr. We could just add a new method that takes an additional arg setupCmd.
2) Shouldn't the addSetupCommand method put the strings in qoutes (single qoutes similar to what is done in addCommand). Also the semi-colon should be probably added after the for-loop terminates. And there should be a space between each of the strings (very similar to what is done in addCommand). Now the question is whether we really require an additional method addSetupCommand ?
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-2765:
--------------------------------------------
Status: Patch Available (was: Open)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2765) wrapper for streaming tasks
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley reassigned HADOOP-2765:
-----------------------------------
Assignee: Amareshwari Sri Ramadasu
> wrapper for streaming tasks
> ---------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
Increasing maxmemory to 1024MB.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
I think contrib tests failed because of maxmemory being 384MB. I'm increaing it to 512MB. Tests passed in my machine with both 384MB and 512MB.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572808#action_12572808 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376389/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1840/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574371#action_12574371 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376954/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 8 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1884/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1884/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1884/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1884/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
Here is a patch to set memory limits for pipes and streaming tasks.
As suggested by Sameer in offline, I didnt introduce any new config item here.
Now Java tasks are still managed by Xmx limit. Pipes and streaming are set memory limit with max memory of parent java task. i.e. Pipes and streaming tasks are started with
ulimit -m Runtime.getRuntime().maxMemory() / 1024
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-2765:
--------------------------------------------
Release Note: This feature enables specifying ulimits for streaming/pipes tasks. Now pipes and streaming tasks have same virtual memory available as the java process which invokes them. Ulimit value will be the same as -Xmx value for java processes provided using mapred.child.java.opts. (was: This feature enables specifying ulimits for streaming/pipes tasks. Now pipes and streaming tasks have same memory available as the java process which invokes them. Ulimit value will be the same as -Xmx value for java processes provided using mapred.child.java.opts.)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-2765:
--------------------------------
Fix Version/s: (was: 0.16.1)
0.17.0
This is deemed too risky for a bug fix release. Deferring to 0.17.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) wrapper for streaming tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564741#action_12564741 ]
Devaraj Das commented on HADOOP-2765:
-------------------------------------
Please submit the patch. definitely interesting.
> wrapper for streaming tasks
> ---------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576895#action_12576895 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377506/2765.1.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 8 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1926/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1926/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1926/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1926/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2765:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2765:
----------------------------------
Fix Version/s: (was: 0.16.1)
Summary: setting memory limits for tasks (was: wrapper for streaming tasks)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) wrapper for streaming tasks
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565137#action_12565137 ]
Runping Qi commented on HADOOP-2765:
------------------------------------
+1
> wrapper for streaming tasks
> ---------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2765) setting memory limits
for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573711#action_12573711 ]
amareshwari edited comment on HADOOP-2765 at 2/29/08 4:56 AM:
---------------------------------------------------------------------------
The patch addresses following issues:
1. The memory limit setting is done by ulimit -v (instead of ulimit -m as in previous patch). Since ulimit -v is the maximum amount of virtual memory available for shell.
2. Now that all streaming tasks get same virtual memory as the parent java task, lauching a java streaming task (eg. TrApp.class) would require more memory than 256MB (set in build-contrib.xml). Noticed this in unit tests. So that value for maxmemory is increased to 384MB in build-contrib.xml. This is required for the existing unit tests for streaming to pass .
3. Added a testcase in contrib/streaming. The test will launch a streaming app which will allocate 10MB memory. First, program is launched with sufficient memory. And test expects it to succeed. Then program is launched with insufficient memory and is expected to be a failure.
was (Author: amareshwari):
The patch addresses following issues:
1. The memory limit setting is done by ulimit -v (instead of ulimit -m as in previous patch). Since ulimit -v is the maximum amount of virtual memory available for shell.
2. Now that all streaming tasks get same virtual memory as the parent java task, lauching a local job runner in streaming task would require more memory than 256MB (set in build-contrib.xml). Noticed this in unit tests. So that value is increased to 384MB. Now the unit tests are fine.
3. Added a testcase in contrib/streaming. The test will launch a streaming app which will allocate 10MB memory. First, program is launched with sufficient memory. And test expects it to succeed. Then program is launched with insufficient memory and is expected to be a failure.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570205#action_12570205 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12375915/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac -1. The applied patch generated 624 javac compiler warnings (more than the trunk's current 623 warnings).
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1815/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1815/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1815/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1815/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
The patch addresses following issues:
1. The memory limit setting is done by ulimit -v (instead of ulimit -m as in previous patch). Since ulimit -v is the maximum amount of virtual memory available for shell.
2. Now that all streaming tasks get same virtual memory as the parent java task, lauching a local job runner in streaming task would require more memory than 256MB (set in build-contrib.xml). Noticed this in unit tests. So that value is increased to 384MB. Now the unit tests are fine.
3. Added a testcase in contrib/streaming. The test will launch a streaming app which will allocate 10MB memory. First, program is launched with sufficient memory. And test expects it to succeed. Then program is launched with insufficient memory and is expected to be a failure.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
It seems like /* @deprecated */ is deprecated. And we have to use @Deprecated at the method starting. Sorry for wrong patch.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) wrapper for streaming tasks
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye updated HADOOP-2765:
-------------------------------------
Fix Version/s: 0.16.1
> wrapper for streaming tasks
> ---------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Fix For: 0.16.1
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Fix Version/s: 0.17.0
Status: Open (was: Patch Available)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
Incorporated Devaraj's comments.
Patch doesnt deprecate captureOutAndErr. and addCommand is used instead of addSetupCommand
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573753#action_12573753 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376819/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 8 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1876/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1876/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1876/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1876/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-2765:
--------------------------------------------
Attachment: 2765.1.patch
Patch, from Devaraj, which fixes tests on windows and MAC.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2765:
--------------------------------
Status: Open (was: Patch Available)
Cancelling patch since this doesn't work on cygwin and MAC.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) wrapper for streaming tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568531#action_12568531 ]
Amareshwari Sri Ramadasu commented on HADOOP-2765:
--------------------------------------------------
In stead of writing C wrapper, we can set memory limit using ulimit from bash where the child process exec happens.
bq. setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
Now instead of Xmx limit, we shall have ulimit value set by "mapred.child.memory". So, this memory limit will be valid for java tasks, pipes and streaming. (since all are execed by bash).
Thoughts?
> wrapper for streaming tasks
> ---------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Fix For: 0.16.1
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570672#action_12570672 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376014/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac -1. The applied patch generated 620 javac compiler warnings (more than the trunk's current 619 warnings).
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs -1. The patch appears to cause Findbugs to fail.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1820/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1820/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1820/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
This patch should fix javac and findbugs warnings.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Patch Available (was: Open)
Sorry for the javadoc warning that was by mistake that was an attempt to get off javac warning.
Even though the method is annotated with @deprecated, I could see the following javac warning:
[javac] src/java/org/apache/hadoop/mapred/TaskLog.java:204: warning: [dep-ann] deprecated name isnt annotated with @Deprecated
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570639#action_12570639 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376006/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc -1. The javadoc tool appears to have generated 1 warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1819/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1819/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1819/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1819/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2765) setting memory limits for tasks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570952#action_12570952 ]
Hadoop QA commented on HADOOP-2765:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376090/patch-2765.txt
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1823/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1823/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1823/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1823/console
This message is automatically generated.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Status: Open (was: Patch Available)
Tests failed even with 512MB. Canceling the patch.
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.16.1
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sri Ramadasu updated HADOOP-2765:
---------------------------------------------
Attachment: patch-2765.txt
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sri Ramadasu
> Fix For: 0.17.0
>
> Attachments: patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2765) setting memory limits for tasks
Posted by "Rick Cox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rick Cox updated HADOOP-2765:
-----------------------------
Hadoop Flags: [Incompatible change]
> setting memory limits for tasks
> -------------------------------
>
> Key: HADOOP-2765
> URL: https://issues.apache.org/jira/browse/HADOOP-2765
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/streaming
> Affects Versions: 0.15.3
> Reporter: Joydeep Sen Sarma
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.17.0
>
> Attachments: 2765.1.patch, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt, patch-2765.txt
>
>
> here's the motivation:
> we want to put a memory limit on user scripts to prevent runaway scripts from bringing down nodes. this setting is much lower than the max. memory that can be used (since most likely these tend to be scripting bugs). At the same time - for careful users, we want to be able to let them use more memory by overriding this limit.
> there's no good way to do this. we can set ulimit in hadoop shell scripts - but they are very restrictive. there doesn't seem to be a way to do a setrlimit from Java - and setting a ulimit means that supplying a higher Xmx limit from the jobconf is useless (the java process will be limited by the ulimit setting when the tasktracker was launched).
> what we have ended up doing (and i think this might help others as well) is to have a stream.wrapper option. the value of this option is a program through which streaming mapper and reducer scripts are execed. in our case, this wrapper is small C program to do a setrlimit and then exec of the streaming job. the default wrapper puts a reasonable limit on the memory usage - but users can easily override this wrapper (eg by invoking it with different memory limit argument). we can use the wrapper for other system wide resource limits (or any environment settings) as well in future.
> This way - JVMs can stick to mapred.child.opts as the way to control memory usage. This setup has saved our ass on many occasions while allowing sophisticated users to use high memory limits.
> Can submit patch if this sounds interesting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.