You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2011/11/04 20:09:51 UTC
[jira] [Created] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Yarn clients / AM should be able to provide config options to the RM / NM
-------------------------------------------------------------------------
Key: MAPREDUCE-3359
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
resolving as per the discussion on this jira.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144318#comment-13144318 ]
Siddharth Seth commented on MAPREDUCE-3359:
-------------------------------------------
bq. I think key value pairs are too generic and can get very confusing to make sense of.
Looking at JobConf and Job - they do already have lots of individual APIs (along with allowing config via a conf object in Job). Yarn RM/NM have explicit APIs only (App POV) - maybe it is better leaving it as is.
The list will just end up growing though - there's 2-3 log params which eventually need to make it through to an NM, maybe other parameters later.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144358#comment-13144358 ]
Hadoop QA commented on MAPREDUCE-3359:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502527/MR-3359.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1250//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1250//console
This message is automatically generated.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Attachment: MR-3359.txt
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Status: Patch Available (was: Open)
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Status: Open (was: Patch Available)
Canceling to upload an updated patch, with more testing.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144297#comment-13144297 ]
Siddharth Seth commented on MAPREDUCE-3359:
-------------------------------------------
This could be a key-value list in the ContainerLaunchContext (ApplicationSubmissionContext would be better but that isn't available to the NM).
An alternate would be to add individual properties - like setLogAggPolicy(), setCancelTokenOnAppComplete. This list would just end up growing though, so prefer the first option.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-3359) Yarn clients / AM
should be able to provide config options to the RM / NM
Posted by "Mahadev konar (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144312#comment-13144312 ]
Mahadev konar edited comment on MAPREDUCE-3359 at 11/4/11 8:28 PM:
-------------------------------------------------------------------
I think key value pairs are too generic and can get very confusing to make sense of. I dont think we'll be adding too many such options to the submission context, so I'd prefer an API to make it explicit and less confusing to use.
was (Author: mahadev):
I think key value pairs are too generic and can get very confusing to make sense of. I dont think we'll be adding too many such options to the launch context, so I'd prefer an API to make it explicit and less confusing to use.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144330#comment-13144330 ]
Daryn Sharp commented on MAPREDUCE-3359:
----------------------------------------
Although I helped contrib to this jira, I have concerns regarding its safety and hope it's a temporary fix.
I feel the config is of questionable value since a misbehaving client may "forget" to cancel its tokens. The NN is holding tokens in memory so it could lead to a potential, and perhaps unintentional, denial of service attack.
When tokens are shared between jobs, it's ambiguous as to when the tokens can be safely cancelled. How does a client know that other running or queued jobs are using the tokens? If the client intends to launch multiple jobs, but the client errors out, the tokens can't be cancelled or "very bad" things will happen to the jobs already submitted. Tasks will pound on the NN every second with the bad token, and yarn tasks appear to run "forever" if rpc connections fail. In a test env, orphaned tasks had pounded on the NN every second for a *month*.
Allowing the RM to cancel tokens when the job completes, which implies tokens are good for one and only one job submission, removes the ambiguity of when it's safe to cancel the tokens. This reduces the chance of a dos attack on the NN, and from a security perspective closes the window of exposure vs. allowing tokens to linger until their lifetime expires.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144391#comment-13144391 ]
Robert Joseph Evans commented on MAPREDUCE-3359:
------------------------------------------------
I will move it over to MAPREDUCE-3291. I will also take a look at the PB code to see if it is getting set correctly or not.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Assignee: Robert Joseph Evans
Target Version/s: 0.23.1
Status: Patch Available (was: Open)
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144356#comment-13144356 ]
Mahadev konar commented on MAPREDUCE-3359:
------------------------------------------
Bobby,
the patch looks fine. A minor nit, we should just add javadoc on setCancelTokensWhenComplete to make sure we warn users not to use it without reason. Something like:
WARNING: this is not recommended unless you want your single job tokens to be reused by others jobs (highly unlikely).
Couldnt come up with something better. Feel free to innovate! :)
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144335#comment-13144335 ]
Hadoop QA commented on MAPREDUCE-3359:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502522/MR-3359.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1249//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1249//console
This message is automatically generated.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144363#comment-13144363 ]
Mahadev konar commented on MAPREDUCE-3359:
------------------------------------------
Also, I think the patch should be filed against MAPREDUCE-3291? no? We can leave this open for any api's for adding to logs or others? Am not too picky abt it, we can mark the other one as duplicate but might want to change the subject of the jira to reflect what its doing. The current subject is very vague and confusing.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144312#comment-13144312 ]
Mahadev konar commented on MAPREDUCE-3359:
------------------------------------------
I think key value pairs are too generic and can get very confusing to make sense of. I dont think we'll be adding too many such options to the launch context, so I'd prefer an API to make it explicit and less confusing to use.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144332#comment-13144332 ]
Robert Joseph Evans commented on MAPREDUCE-3359:
------------------------------------------------
I think that protocol buffers offers some benefits over generic key/value pairs. It includes types, even complex types that are very difficult to do with just key/value pairs. It helps to ensure binary compatibility and has default values built in. Yes if we do remove a config value that number must remain reserved for eternity and if we add in a new config it must be added to the proto file. Keeping a number reserved is not that painful and after having gone through the process of trying to document/clean up all of the yarn config options that where not published in the yarn-defaults.xml I am 100% in favor of forcing a single machine readable location for parameter values.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144352#comment-13144352 ]
Siddharth Seth commented on MAPREDUCE-3359:
-------------------------------------------
Took a quick look at the patch. The PB part needs a bit of change - doesn't look like the set call will ever make it back to the proto object. Ref NodeHealthStatusPBImpl.setIsNodeHealth()
Will continue looking.
We should just move this to 3291 which is what this patch is fixing, and close out this jira since we're not adding a k-v pair to pass configuration to the NM/RM.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3359) Yarn clients / AM should be able
to provide config options to the RM / NM
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3359:
-------------------------------------------
Attachment: MR-3359.txt
This patch adds in support for mapreduce.job.complete.cancel.delegation.tokens, which was left out of MRV2. I have not added in any new tests yet to verify the code works as expected. This is mostly to get feedback on the patch.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Attachments: MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144365#comment-13144365 ]
Mahadev konar commented on MAPREDUCE-3359:
------------------------------------------
Damn! didnt read Sid's comments :).
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be
able to provide config options to the RM / NM
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144345#comment-13144345 ]
Mahadev konar commented on MAPREDUCE-3359:
------------------------------------------
Daryn,
Very valid concern. We should have a broader discussion on it. For now for keeping things backwards compatible, I think Bobby's approach should be fine.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-3359
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Robert Joseph Evans
> Attachments: MR-3359.txt, MR-3359.txt
>
>
> The RM and NM do not read a job's configuration. Clients / AMs should however be able to configure certain parameters for the RM/NM on a per app basis - like the Log Retention policy, token cancellation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira