You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2011/09/08 06:25:09 UTC
[jira] [Created] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
GH-26: add support for multiple workflow XMLs in a single HDFS directory
------------------------------------------------------------------------
Key: OOZIE-72
URL: https://issues.apache.org/jira/browse/OOZIE-72
Project: Oozie
Issue Type: Bug
Reporter: Hadoop QA
Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
In many cases is desirable to share configurations and binaries among multiple workflow apps.
Today this is not possible.
Proposal:
1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101769#comment-13101769 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
Here is the latest patch that can work for both workflow and coordinator apps.
Test cases are cleaned up along the way.
http://github.com/brookwc/oozie/commit/9d162fc4aa44190bb9f5554aa10dfb3a219c1a06
It has some significant change from last time, please review. Will create a pull request after review passes here.
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101766#comment-13101766 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
Patch is here. Please review.
http://github.com/brookwc/oozie/commit/df33a61f610105b83564fb5cbb046695c882278c
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101764#comment-13101764 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
tucu00 remarked:
brookwc,
Great, a couple of things:
1* Unless you are a Yahoo employee, you'll need to have a CLA submitted to Yahoo in order for the patch to be accepted.
2* Do you have an estimate for the patch?
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099768#comment-13099768 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
A question is that how do you differentiate if path points to a file or a directory.
For example, I can have the following:
path = /a/b/workflow.xml
This path is ambiguous, it could be pointing to a workflow.xml under /a/b/ or /a/b/workflow.xml/.
I checked /a/b/workflow.xml/ is a valid hdfs path.
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101762#comment-13101762 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
tucu00 remarked:
My idea is that the disambiguation rule is:
1* if path is DIR, then look for workflow in workflow.xml and the parent is the app root
2* if path is a file, then the file is the workflow and the parent is the app root
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (OOZIE-72) GH-26: add support for multiple workflow
XMLs in a single HDFS directory
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik closed OOZIE-72.
---------------------------------
Resolution: Fixed
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101763#comment-13101763 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
This looks good to me.
I can work on this small task.
Thanks.
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101770#comment-13101770 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
bansalmayank remarked:
Closed by fbab0ab06f64467d6f709590ece77adea3c76844 add support for multiple workflow XMLs in a single HDFS
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101765#comment-13101765 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
1. Yes, I am Yahoo employee.
2. Patch will be ready for review in a day.
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101768#comment-13101768 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
tucu00 remarked:
this should work also for coordinator apps
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101767#comment-13101767 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
brookwc remarked:
This is a new patch according to tucu00's review feedback. thanks tucu00!
http://github.com/brookwc/oozie/commit/32536b4a873ea27440fd8f97d9254fad85b610a9
Still some test cases (around 30-40) need to be cleaned up accordingly (should be mostly mechanical).
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-72) GH-26: add support for multiple
workflow XMLs in a single HDFS directory
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101771#comment-13101771 ]
Hadoop QA commented on OOZIE-72:
--------------------------------
mikelikespie remarked:
The documentation should be updated to reflect this change.
> GH-26: add support for multiple workflow XMLs in a single HDFS directory
> ------------------------------------------------------------------------
>
> Key: OOZIE-72
> URL: https://issues.apache.org/jira/browse/OOZIE-72
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> Currently a workflow XML is the 'workflow.xml' file under the HDFS directory specified in the job property 'oozie.wf.application.path'.
> This means that a given HDFS directory can have only one workflow app (the workflow.xml file).
> In many cases is desirable to share configurations and binaries among multiple workflow apps.
> Today this is not possible.
> Proposal:
> 1* If 'oozie.wf.application.path' points to a HDFS directory, the workflow app is 'workflow.xml' (today's behavior)
> 2* If 'oozie.wf.application.path' points to an XML file in HDFS, the workflow app is the specified file path and the workflow app directory (for all resources and and binaries) is the parent directory.
> This proposal preserves backwards compatibility.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira