You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Robert Kanter (JIRA)" <ji...@apache.org> on 2012/07/17 01:41:33 UTC
[jira] [Created] (OOZIE-913) Add Name Node Element to FS action
Robert Kanter created OOZIE-913:
-----------------------------------
Summary: Add Name Node Element to FS action
Key: OOZIE-913
URL: https://issues.apache.org/jira/browse/OOZIE-913
Project: Oozie
Issue Type: Improvement
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Minor
Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times.
e.g.
{code}
<action name="fs-node">
<fs>
<name-node>hdfs://host:port</name-node>
<mkdir path="/user/${wf:user()}/output-data/1"/>
<mkdir path="/user/${wf:user()}/output-data/2"/>
</fs>
<ok to="end"/>
<error to="fail"/>
</action>
{code}
This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Mona Chitnis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425271#comment-13425271 ]
Mona Chitnis commented on OOZIE-913:
------------------------------------
Hi Robert,
Yes 477 is for schemes. I had missed seeing in your example that your scheme is the same, sorry about that. The approach you propose can be a separate JIRA I believe.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Mona Chitnis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424025#comment-13424025 ]
Mona Chitnis commented on OOZIE-913:
------------------------------------
Why is a new Filesystem object initialized for each file path that FsAE performs an operation on? The DFS initialize method that it calls internally only needs the NN scheme, authority, and the parent dir /user/<username> dir where all the jobs data for a user resides.
This way, we dont need to pass argument fsConf to each fs operation chmod, del, mkdir etc, and can initialize the fs object one-time before doOperations() itself.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421783#comment-13421783 ]
Alejandro Abdelnur commented on OOZIE-913:
------------------------------------------
* FsActionExecutor.validatePathWithNameNode()
By seeing where this method is being used, shouldn't a better name for this method be resolveToFullPath() or something like that?
There should be a single return at the end of method, then is easier to follow the method flow.
Not sure if we should support relative path. I think they should be always absolute. For WF app files relative means within the WF app directory. But I doubt that for data users would be doing stuff within the WF app directory.
* FsActionExecutor.doOperations()
Why are we parsing the full JobXML again? are you after the global configuration? if so, the action protoconfiguration available thru the context should have all those properties.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Mona Chitnis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425156#comment-13425156 ]
Mona Chitnis commented on OOZIE-913:
------------------------------------
Yes we are going to allow multiple filesystems. However, the user has to configure his server to 'support' that filesystem (OOZIE-477), so we avoid users specifying NFS or local filesystem in path. Hence the FS ActionExecutor could access this property before going to doOperations and initialize the fs objects one time for each supported fs.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node Element to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420778#comment-13420778 ]
Robert Kanter commented on OOZIE-913:
-------------------------------------
Currently, the FS instance is created based on the name node and the user. What things can be put into a <job-xml> or <configuration> element that affect how the FS instance is created?
> Add Name Node Element to FS action
> ----------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427456#comment-13427456 ]
Alejandro Abdelnur commented on OOZIE-913:
------------------------------------------
patch looks good, a comments though:
*FSActionExecutor: patch introduces unused imports
*FSActionExecutor, resolveToFullPath(): when the path is absolute, we should use the namenode scheme&authority to create the path, not the full namenode path (in case somebody specifies as their name-node hdfs://foo:1000/bar).
*FSActionExecutor, resolveToFullPath(): If I'm not mistaken, pathScheme + pathAuthority will produce 'hdfsHOST:PORT', not 'hdsf://HOST:PORT. Same as previous comment on the namenode path
*FSActionExecutor, parseJobXML() & parseConfiguration(): are this methods duplicated from other place? I believe so. Shouldn't they belong to JavaActionExecutor and everybody consume it from there?
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-913:
--------------------------------
Attachment: OOZIE-913.patch
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422499#comment-13422499 ]
Robert Kanter commented on OOZIE-913:
-------------------------------------
Do you mean the the file(s) referenced in the <job-xml> element(s)? That's only looked at by the JavaActionExecutor and its subclasses; fsActionExecutor is actually a direct subclass of ActionExecutor so <job-xml> won't be parsed otherwise.
I'm pretty sure the same is true for the <name-node> etc. I dumped out what was in the protoconfiguration and it didn't have any of these already.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425196#comment-13425196 ]
Robert Kanter commented on OOZIE-913:
-------------------------------------
My understanding of OOZIE-477 is that you can check with a list of supported filesystem types (i.e. schemes), so that you can use other filesystems besides hdfs, as long as its on the list. But a FileSystem object isn't just a scheme (e.g. hdfs://), its also an authority (e.g. hdfs://host1:port). So, in my previous example, its true that they are both hdfs filesystems, but to access both of them, you'd need two FileSystem objects. Correct me if I'm wrong, but it doesn't have a list of actual existing filesystems; only the supported types. We also don't necessarily want to initialize FS objects for each supported fs (or actual existing filesystem) because the action might not be using all of them.
If we wanted to minimize the number of times FsAE initializes FS objects, we could have it read through the XML before doing any actual FS actions to build a queue of FS actions and a list of all FS's used by these actions so we can initialize each FS only once and then use those FS objects to execute the queue of actions; this would require some fundamental changes in how the FsAE works, and is maybe better off as its own JIRA?
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-913:
--------------------------------
Description:
Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
e.g.
{code}
<action name="fs-node">
<fs>
<name-node>hdfs://host:port</name-node>
<job-xml>fs-info.xml</job-xml>
<configuration>
<property>
<name>some.property</name>
<value>some.value</value>
</property>
</configuration>
<mkdir path="/user/${wf:user()}/output-data/1"/>
<mkdir path="/user/${wf:user()}/output-data/2"/>
</fs>
<ok to="end"/>
<error to="fail"/>
</action>
{code}
This can then also leverage the global section from OOZIE-874 automatically.
was:
Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times.
e.g.
{code}
<action name="fs-node">
<fs>
<name-node>hdfs://host:port</name-node>
<mkdir path="/user/${wf:user()}/output-data/1"/>
<mkdir path="/user/${wf:user()}/output-data/2"/>
</fs>
<ok to="end"/>
<error to="fail"/>
</action>
{code}
This can then also leverage the global section from OOZIE-874 automatically.
Summary: Add Name Node, job-xml, and configuration Elements to FS action (was: Add Name Node Element to FS action)
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424963#comment-13424963 ]
Robert Kanter commented on OOZIE-913:
-------------------------------------
That's the way it worked before my changes; I think that it's allowed (though maybe uncommon) to use different filesystems in the same FS action, in which case we can't initialize the fs object one-time. Something like this:
{code}
<action name="fs-node">
<fs>
<mkdir path="hdfs://host1:port/dir"/>
<mkdir path="hdfs://host2:port/dir"/>
</fs>
<ok to="end"/>
<error to="fail"/>
</action>
{code}
If that's not allowed, then I agree that a single fs object makes sense.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node Element to FS action
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415795#comment-13415795 ]
Alejandro Abdelnur commented on OOZIE-913:
------------------------------------------
we should support a <job-xml> & <configuration> elements which would be use to create the FS instance.
> Add Name Node Element to FS action
> ----------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-913:
--------------------------------
Attachment: OOZIE-913-v3.patch
Attached v3 of patch:
- Removed unused imports
- Fixed the issue with the full namenode path and added updated a test
- Fixed the pathScheme + pathAuthority issue
Together, FsActionExecutor.parseJobXML() and FsActionExecutor.parseConfiguration are very similar to JavaActionExecutor.setupActionConf(). Most of the executors inherit from JavaActionExecutor, so they get this parsing for free; but fsActionExecutor does not and setupActionConf() is not static.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913-v3.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427681#comment-13427681 ]
Alejandro Abdelnur commented on OOZIE-913:
------------------------------------------
+1. Robert, would you please file JIRA to remove the duplication? it seems there are some additional changes in doing that that are outside of the scope of this JIRA.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913-v3.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated OOZIE-913:
--------------------------------
Attachment: OOZIE-913-v2.patch
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-913) Add Name Node, job-xml, and
configuration Elements to FS action
Posted by "Robert Kanter (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423193#comment-13423193 ]
Robert Kanter commented on OOZIE-913:
-------------------------------------
The <job-xml> that is being parsed is just the one from the Fs action.
> Add Name Node, job-xml, and configuration Elements to FS action
> ---------------------------------------------------------------
>
> Key: OOZIE-913
> URL: https://issues.apache.org/jira/browse/OOZIE-913
> Project: Oozie
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Priority: Minor
> Attachments: OOZIE-913-v2.patch, OOZIE-913.patch
>
>
> Adding a name node Element to the FS action will allow users to shorten FS actions and avoid specifying the name node (hdfs://HOST:PORT) multiple times. We can also add job-xml and configuration elements to allow users to set properties when the FS instance is created.
> e.g.
> {code}
> <action name="fs-node">
> <fs>
> <name-node>hdfs://host:port</name-node>
> <job-xml>fs-info.xml</job-xml>
> <configuration>
> <property>
> <name>some.property</name>
> <value>some.value</value>
> </property>
> </configuration>
> <mkdir path="/user/${wf:user()}/output-data/1"/>
> <mkdir path="/user/${wf:user()}/output-data/2"/>
> </fs>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> {code}
> This can then also leverage the global section from OOZIE-874 automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira