You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2008/04/04 15:17:24 UTC
[jira] Created: (HADOOP-3171) speculative execution should not have
default value on hadoop-default.xml bundled in the Hadoop JAR
speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
---------------------------------------------------------------------------------------------------
Key: HADOOP-3171
URL: https://issues.apache.org/jira/browse/HADOOP-3171
Project: Hadoop Core
Issue Type: Bug
Components: conf
Affects Versions: 0.16.2
Environment: all
Reporter: Alejandro Abdelnur
Fix For: 0.17.0
Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
*ON in hadoop-default.xml (current behavior)*
* ON in JT hadoop-site.xml
* present in job.xml, job's value is used
* not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
* ON FINAL in the JT hadoop-site.xml
* present or not present in the job.xml, ON is used
* OFF in JT hadoop-site.xml
* present in job.xml, job's value is used
* not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
* OF FINAL in the JT hadoop-site.xml
* present or not present in the job.xml, OFF is used
*OFF in hadoop-default.xml (not current behavior)*
* ON in JT hadoop-site.xml
* present in job.xml, job's value is used
* not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
* ON FINAL in the JT hadoop-site.xml
* present or not present in the job.xml, ON is used
* OFF in JT hadoop-site.xml
* present in job.xml, job's value is used
* not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
* OF FINAL in the JT hadoop-site.xml
* present or not present in the job.xml, OFF is used
---
Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-3171.
-----------------------------------
Resolution: Won't Fix
-1 on Arun's patch because the first resource is not distinguished as the default resource. Not saving it in the serialization will lead to confusion and delay.
If a cluster admin wants to force speculation off, they just need to set it off on the job tracker as a final parameter. This is not a bug, but a configuration problem..
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Subramaniam Krishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586335#action_12586335 ]
Subramaniam Krishnan commented on HADOOP-3171:
----------------------------------------------
I think a simpler solution would be that the Job Client does NOT load the hadoop-default.xml , instead it only loads the hadoop-site.xml if it's present.
This way, only the default values on Server side take effect.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Fix For: 0.17.0
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned HADOOP-3171:
-------------------------------------
Assignee: Arun C Murthy
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3171) speculative execution should not have
default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-3171:
--------------------------------
Fix Version/s: (was: 0.17.0)
Not a blocker for Hadoop 0.17.0
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587606#action_12587606 ]
Owen O'Malley commented on HADOOP-3171:
---------------------------------------
I left off:
4. Use the tool interface that lets you define properties and config files on the command line.
Either the command line or the environment variable should work for you.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587506#action_12587506 ]
Arun C Murthy commented on HADOOP-3171:
---------------------------------------
The intention was to ensure that we only serialize non-default values (i.e. values in hadoop-site.xml) via Configuration.write. This would ensure that the right behaviour occured...
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved HADOOP-3171.
----------------------------------------
Resolution: Won't Fix
Closing this issue as it is not reflect the real problem, I'll open another one.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur reopened HADOOP-3171:
----------------------------------------
If a cluster wants to force spec-exec (or any other prop) to a value can set it as final.
If a cluster wants to set spec-exec (or any other prop) to a default value for that cluster, this is not possible (it is only possible to do this for the hadoop-default.xml value).
IMO, I should be able to set the default value I want my cluster to use if the property is not set, and this is not supported now.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587562#action_12587562 ]
Owen O'Malley commented on HADOOP-3171:
---------------------------------------
I don't see the use case for having the cluster pushing a default for speculative execution. You can either:
1. You can build a new client jar that has the default changed.
2. You can force the value from the server.
3. The client can define a HADOOP_CONF_DIR with a hadoop-site.xml that changes the default.
None of this should reasonably change depending on what server you connect to. It is *not* a property of a given server.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3171) speculative execution should not have
default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-3171:
----------------------------------
Attachment: HADOOP-3171_20080410.patch
Hmm... untested patch.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3171) speculative execution should not
have default value on hadoop-default.xml bundled in the Hadoop JAR
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587571#action_12587571 ]
Alejandro Abdelnur commented on HADOOP-3171:
--------------------------------------------
On #1
Asking people to rejar the hadoop.jar is not reasonable as it may lead to nasty debug situations.
On #2
spec exec is an example only (in our case most of our jobs cannot run with spec-exec but a few, so we want to set it off by default, and only set it on in the few jobs that we want it on).
Take the case of number of task retries, replication factor, compression or compression-type properties. You may want to set the default behavior of your cluster, but not make them final.
On #3
It is not always possible to have a HADOOP_CONF_DIR for example from within a webApp. It gets complicated.
---
And if I have a hadoop client that dispatches jobs to different clusters things may get even more complicated.
My concern is that a cluster should be able to control what a default value is without forcing with a final. This is not possible today.
> speculative execution should not have default value on hadoop-default.xml bundled in the Hadoop JAR
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3171
> URL: https://issues.apache.org/jira/browse/HADOOP-3171
> Project: Hadoop Core
> Issue Type: Bug
> Components: conf
> Affects Versions: 0.16.2
> Environment: all
> Reporter: Alejandro Abdelnur
> Assignee: Arun C Murthy
> Attachments: HADOOP-3171_20080410.patch
>
>
> Having a default value for speculative execution in the hadoop-default.xml bundled in the Hadoop JAR file does not allow a cluster to control the default behavior.
> *ON in hadoop-default.xml (current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> *OFF in hadoop-default.xml (not current behavior)*
> * ON in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, OFF is taken as default from the hadoop-default.xml present in the client's JAR/conf (*)
> * ON FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, ON is used
> * OFF in JT hadoop-site.xml
> * present in job.xml, job's value is used
> * not-present in job.xml, ON is taken as default from the hadoop-default.xml present in the client's JAR/conf
> * OF FINAL in the JT hadoop-site.xml
> * present or not present in the job.xml, OFF is used
> ---
> Still is desirable for the JT to have a default value. To avoid having to support 2 hadoop-default.xml files, one for the JT and other for the clients, the easiest why is to remove it from the hadoop-default.xml and have the default value in the code when getting the config property (thing that may be already happening).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.