You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Sean Mackrory (JIRA)" <ji...@apache.org> on 2013/05/03 01:10:17 UTC
[jira] [Comment Edited] (BIGTOP-882) Upload content of Oozie
sharelib to HDFS
[ https://issues.apache.org/jira/browse/BIGTOP-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13648010#comment-13648010 ]
Sean Mackrory edited comment on BIGTOP-882 at 5/2/13 11:09 PM:
---------------------------------------------------------------
Had a quick offline conversation about some options. I'll do my best to recreate the details here - forgive any clumsiness as this is my first exposure to this issue. There are a few options:
* Using init-hdfs.sh in its current form
* Using Oozie's Java tools to upload the generated tarball (or a directory listing all the libraries)
The tarball may not contain the correct versions initially, and will cease to contain the correct versions when other components are updated (unless they rebuild Oozie - not reasonable). We could generate a directory containing symlinks to the current libraries, and then use Oozie's tool to use that, but this doesn't offer any clear benefits over using init-hdfs in the first place. The main difference would be that init-hdfs.sh doesn't require you to enter the fs url, but it also doesn't support the property in Oozie that specifies arbitrary locations for the sharelib. Given all that, we decided that the cleanest solution for Bigtop right now was to continue using init-hdfs.sh in its current form. If users install or upgrade new components, they need only run the script again to sync their sharelib. I'm certainly open to improving this in the future if Oozie changes the way this is managed on the server, but I think this is the all-around best option right now.
If I'm missing any details or you have anything else to add, please speak up, otherwise I'll re-close this as resolved.
was (Author: mackrorysd):
Had a quick offline conversation about some options. I'll do my best to recreate the details here - forgive any clumsiness as this is my first exposure to this issue. There are a few options:
* Using init-hdfs.sh in its current form
* Using Oozie's Java tools to upload the generated tarball (or a directory listing all the libraries)
The tarball may not contain the correct versions initially, and will cease to contain the correct versions when other components are updated (unless they rebuild Oozie - not reasonable). We could generate a directory containing symlinks to the current libraries, and then use Oozie's tool to use that, but this doesn't offer any clear benefits over using init-hdfs in the first place. The main difference would be that init-hdfs.sh doesn't require you to enter the fs url, but it also doesn't support the property in Oozie that specifies arbitrary locations for the sharelib. Given all that, we decided that the cleanest solution for Bigtop right now was to continue using init-hdfs.sh in its current form. If users install or upgrade new components, they need only run the script again to sync their sharelib. I'm certainly open to improving this in the future if Oozie changes the way this is managed on the server, but I think this is the all-around best option right now.
If I'm missing any details or you have anything else to add, please speak up, otherwise I'll re-close this as resolved.
> Upload content of Oozie sharelib to HDFS
> ----------------------------------------
>
> Key: BIGTOP-882
> URL: https://issues.apache.org/jira/browse/BIGTOP-882
> Project: Bigtop
> Issue Type: Improvement
> Reporter: Robert Kanter
> Assignee: Sean Mackrory
>
> OOZIE-1054 allows Oozie to automatically upload the sharelib to HDFS; Bigtop should either use that mechanism or do something similar.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira