You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Alejandro Abdelnur (Created) (JIRA)" <ji...@apache.org> on 2012/03/05 23:11:57 UTC
[jira] [Created] (OOZIE-736) Add support for configurations per
JT/NN
Add support for configurations per JT/NN
----------------------------------------
Key: OOZIE-736
URL: https://issues.apache.org/jira/browse/OOZIE-736
Project: Oozie
Issue Type: New Feature
Components: core
Affects Versions: 3.2.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Priority: Critical
Fix For: 3.2.0
Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223125#comment-13223125 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5636
-----------------------------------------------------------
Ship it!
+1 looks good
- Mayank
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223828#comment-13223828 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java, line 193
bq. > <https://reviews.apache.org/r/4186/diff/2/?file=88523#file88523line193>
bq. >
bq. > so will it be an empty configuration? Why not new Configuration() which could add the default hadoop configs.
bq.
bq. Alejandro Abdelnur wrote:
bq. And empty configuration to load the hadoop configs, yes, but those values are injected as defaults into fully populated JobConf/Configuration instances.
bq.
bq. Mohammad Islam wrote:
bq. I am asking the configuration provided in hadoop config files such as (*-default.xml) that come with hadoop library.
bq.
that is loaded in an existing configuration. Stil i'm looking at some issues with the current approach.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223774#comment-13223774 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java, line 193
bq. > <https://reviews.apache.org/r/4186/diff/2/?file=88523#file88523line193>
bq. >
bq. > so will it be an empty configuration? Why not new Configuration() which could add the default hadoop configs.
bq.
bq. Alejandro Abdelnur wrote:
bq. And empty configuration to load the hadoop configs, yes, but those values are injected as defaults into fully populated JobConf/Configuration instances.
I am asking the configuration provided in hadoop config files such as (*-default.xml) that come with hadoop library.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "Mohammad Kamrul Islam (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222723#comment-13222723 ]
Mohammad Kamrul Islam commented on OOZIE-736:
---------------------------------------------
so hadoop configuration file will have these key value pairs. right?
Considering if we could make it more generic. For example, if hive comes up with their cluster specific conf, how could we extend this. or we will write another file.
In short, the proposal is to consider the
same file for multiple cluster configurations irrespective of product (i.e. hadoop, hive )
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222816#comment-13222816 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/
-----------------------------------------------------------
Review request for oozie.
Summary
-------
hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
The Hadoop configuration file would be looked in the Oozie conf/ directory
An example of this configuration would be:
hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
Documentation is still missing, coming soon.
This addresses bug OOZIE-736.
https://issues.apache.org/jira/browse/OOZIE-736
Diffs
-----
trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297298
trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297298
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297298
trunk/core/src/main/resources/oozie-default.xml 1297298
trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297298
trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
Diff: https://reviews.apache.org/r/4186/diff
Testing
-------
run all testcases with Hadoop 1
Thanks,
Alejandro
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223270#comment-13223270 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > Hi Alejandro,
bq. > So how/who will the hadoop-config.xml be populated? Because it is expected to be cluster specific configurations defined in *-site.xml (such as core, hdfs etc).
bq. > Is it we have to create another file? Or we can **reuse** the cluster specified *-site.xml as it is. Only thing we have to configure those file names into oozie-site.xml.
bq. > right?
bq. >
bq. > Regards,
bq. > Mohammad
bq. >
bq. >
Mohammad,
you have to create a hadoop-config.xml file per cluster, as you indicated typically will have the information the *-site.xml files of the cluster used by the clients. You could reuse the cluster specific ones by renaming them, still you may have to combine the core/hdfs/mapred/yarn. And yes, the only thing is to configure them in the the oozie-site.xml and copy them into the oozie conf/ dir.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222664#comment-13222664 ]
Alejandro Abdelnur commented on OOZIE-736:
------------------------------------------
A solution would be to add to the HadoopAccessorService a new configuration property
hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
The Hadoop configuration file would be looked in the Oozie conf/ directory
An example of this configuration would be:
hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223776#comment-13223776 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > Hi Alejandro,
bq. > So how/who will the hadoop-config.xml be populated? Because it is expected to be cluster specific configurations defined in *-site.xml (such as core, hdfs etc).
bq. > Is it we have to create another file? Or we can **reuse** the cluster specified *-site.xml as it is. Only thing we have to configure those file names into oozie-site.xml.
bq. > right?
bq. >
bq. > Regards,
bq. > Mohammad
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammad,
bq.
bq. you have to create a hadoop-config.xml file per cluster, as you indicated typically will have the information the *-site.xml files of the cluster used by the clients. You could reuse the cluster specific ones by renaming them, still you may have to combine the core/hdfs/mapred/yarn. And yes, the only thing is to configure them in the the oozie-site.xml and copy them into the oozie conf/ dir.
can't we use them by reusing the file as it is without renaming and without combining?
This is another level of task and indirection.
Renaming + combining could be handled through some configuration in oozie-site.xml.
Because there will be an impact on us.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223275#comment-13223275 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java, line 193
bq. > <https://reviews.apache.org/r/4186/diff/2/?file=88523#file88523line193>
bq. >
bq. > so will it be an empty configuration? Why not new Configuration() which could add the default hadoop configs.
And empty configuration to load the hadoop configs, yes, but those values are injected as defaults into fully populated JobConf/Configuration instances.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "Alejandro Abdelnur (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved OOZIE-736.
--------------------------------------
Resolution: Fixed
Committed to trunk
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224148#comment-13224148 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > Hi Alejandro,
bq. > So how/who will the hadoop-config.xml be populated? Because it is expected to be cluster specific configurations defined in *-site.xml (such as core, hdfs etc).
bq. > Is it we have to create another file? Or we can **reuse** the cluster specified *-site.xml as it is. Only thing we have to configure those file names into oozie-site.xml.
bq. > right?
bq. >
bq. > Regards,
bq. > Mohammad
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammad,
bq.
bq. you have to create a hadoop-config.xml file per cluster, as you indicated typically will have the information the *-site.xml files of the cluster used by the clients. You could reuse the cluster specific ones by renaming them, still you may have to combine the core/hdfs/mapred/yarn. And yes, the only thing is to configure them in the the oozie-site.xml and copy them into the oozie conf/ dir.
bq.
bq. Mohammad Islam wrote:
bq. can't we use them by reusing the file as it is without renaming and without combining?
bq. This is another level of task and indirection.
bq. Renaming + combining could be handled through some configuration in oozie-site.xml.
bq. Because there will be an impact on us.
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammad,
bq.
bq. Are you suggesting that instead pointing to a file we should point to a directory and from there load all the hadoop *-site.xml files ?
bq.
bq. We could do that, yes it would be simpler for admins.
bq.
bq. My concern here is that we would have to be aware from Oozie side on the order of loading the *-site.xml files. That is why I've thought not to do it.
bq.
bq. But you have point on usability.
bq.
bq. Thoughts?
bq.
I have some idea. will share with you
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223822#comment-13223822 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
bq. On 2012-03-06 10:13:38, Mohammad Islam wrote:
bq. > Hi Alejandro,
bq. > So how/who will the hadoop-config.xml be populated? Because it is expected to be cluster specific configurations defined in *-site.xml (such as core, hdfs etc).
bq. > Is it we have to create another file? Or we can **reuse** the cluster specified *-site.xml as it is. Only thing we have to configure those file names into oozie-site.xml.
bq. > right?
bq. >
bq. > Regards,
bq. > Mohammad
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Mohammad,
bq.
bq. you have to create a hadoop-config.xml file per cluster, as you indicated typically will have the information the *-site.xml files of the cluster used by the clients. You could reuse the cluster specific ones by renaming them, still you may have to combine the core/hdfs/mapred/yarn. And yes, the only thing is to configure them in the the oozie-site.xml and copy them into the oozie conf/ dir.
bq.
bq. Mohammad Islam wrote:
bq. can't we use them by reusing the file as it is without renaming and without combining?
bq. This is another level of task and indirection.
bq. Renaming + combining could be handled through some configuration in oozie-site.xml.
bq. Because there will be an impact on us.
bq.
bq.
bq.
Mohammad,
Are you suggesting that instead pointing to a file we should point to a directory and from there load all the hadoop *-site.xml files ?
We could do that, yes it would be simpler for admins.
My concern here is that we would have to be aware from Oozie side on the order of loading the *-site.xml files. That is why I've thought not to do it.
But you have point on usability.
Thoughts?
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223137#comment-13223137 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/#review5637
-----------------------------------------------------------
Hi Alejandro,
So how/who will the hadoop-config.xml be populated? Because it is expected to be cluster specific configurations defined in *-site.xml (such as core, hdfs etc).
Is it we have to create another file? Or we can **reuse** the cluster specified *-site.xml as it is. Only thing we have to configure those file names into oozie-site.xml.
right?
Regards,
Mohammad
trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java
<https://reviews.apache.org/r/4186/#comment12272>
so will it be an empty configuration? Why not new Configuration() which could add the default hadoop configs.
- Mohammad
On 2012-03-06 05:39:54, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4186/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-06 05:39:54)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
bq.
bq. Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
bq.
bq. In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
bq.
bq. The Hadoop configuration file would be looked in the Oozie conf/ directory
bq.
bq. An example of this configuration would be:
bq.
bq. hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
bq.
bq. Documentation is still missing, coming soon.
bq.
bq.
bq. This addresses bug OOZIE-736.
bq. https://issues.apache.org/jira/browse/OOZIE-736
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
bq. trunk/core/src/main/resources/oozie-default.xml 1297338
bq. trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
bq. trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
bq. trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
bq. trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
bq. trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
bq. trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
bq. trunk/docs/src/site/twiki/index.twiki 1297338
bq.
bq. Diff: https://reviews.apache.org/r/4186/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. run all testcases with Hadoop 1
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-736) Add support for configurations per
JT/NN
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223003#comment-13223003 ]
jiraposter@reviews.apache.org commented on OOZIE-736:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4186/
-----------------------------------------------------------
(Updated 2012-03-06 05:39:54.418868)
Review request for oozie.
Changes
-------
adding documentation
Summary
-------
hadoop.configurations= [<AUTHORITY>=<CONFIG_FILE>]*
Where <AUTHORITY> is the HOST:PORT of the Hadoop JT/NN and <CONFIG_FILE> is the Hadoop configuration file to load from the local filesystem with the defaults for the corresponding JT/NN.
In addition, a '*' wildcard would be support for the <AUTHORITY>, this would be used as default if the job used authority does not have a custom config.
The Hadoop configuration file would be looked in the Oozie conf/ directory
An example of this configuration would be:
hadoop.configuration= *=hadoop-defaultconf.xml myjobtracker:8021=hadoop-cluster1.xml, mynamenode:8020=hadoop-cluster1.xml
Documentation is still missing, coming soon.
This addresses bug OOZIE-736.
https://issues.apache.org/jira/browse/OOZIE-736
Diffs (updated)
-----
trunk/core/src/main/conf/hadoop-config.xml PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 1297338
trunk/core/src/main/resources/oozie-default.xml 1297338
trunk/core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 1297338
trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1297338
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1297338
trunk/core/src/test/resources/hadoop-config.xml PRE-CREATION
trunk/core/src/test/resources/test-hadoop-config.xml PRE-CREATION
trunk/docs/src/site/twiki/AG_HadoopConfiguration.twiki PRE-CREATION
trunk/docs/src/site/twiki/DG_UsingHadoopKerberos.twiki 1297338
trunk/docs/src/site/twiki/index.twiki 1297338
Diff: https://reviews.apache.org/r/4186/diff
Testing
-------
run all testcases with Hadoop 1
Thanks,
Alejandro
> Add support for configurations per JT/NN
> ----------------------------------------
>
> Key: OOZIE-736
> URL: https://issues.apache.org/jira/browse/OOZIE-736
> Project: Oozie
> Issue Type: New Feature
> Components: core
> Affects Versions: 3.2.0
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently Oozie does not support oozie-server-side configs for the JT/NN and users must add all values explicitly in their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira