You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Mark Grover (JIRA)" <ji...@apache.org> on 2013/02/24 09:12:13 UTC
[jira] [Comment Edited] (BIGTOP-852) Improve HDFS helper script
[ https://issues.apache.org/jira/browse/BIGTOP-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585324#comment-13585324 ]
Mark Grover edited comment on BIGTOP-852 at 2/24/13 8:11 AM:
-------------------------------------------------------------
Thanks, cos.
I found another problem, /usr/lib/hive/lib directory contains a symlink:
hbase.jar -> ../../hbase/hbase.jar
This symlink exists because of Hive-HBase integration, however, when Hive is installed without HBase, this link is broken.
Now, the helper script tries to copy over binaries from /usr/lib/hive/lib to oozie's HDFS dir by doing
{code}
if ls /usr/lib/hive/lib/*.jar &> /dev/null; then
sudo -u hdfs hadoop fs -put /usr/lib/hive/lib/*.jar /user/oozie/share/lib/hive
fi
{code}
However, this breaks due to the broken symlinks, breaking the helper script prematurely since it has a {{set -ex}} on the top.
was (Author: mgrover):
Thanks, cos.
I found another problem, /usr/lib/hive/lib directory contains a symlink:
hbase.jar -> ../../hbase/hbase.jar
This symlink exists because of Hive-HBase integration, however, when Hive is installed without HBase, this link is broken.
Now, the helper script tries to copy over binaries from /usr/lib/hive/lib to oozie's HDFS dir by doing
{code}
if ls /usr/lib/hive/lib/*.jar &> /dev/null; then
sudo -u hdfs hadoop fs -put /usr/lib/hive/lib/*.jar /user/oozie/share/lib/hive
fi
{code}
However, this breaks due to the broken symlinks, breaking the helper script prematurely since it has a {set -ex} on the top.
> Improve HDFS helper script
> --------------------------
>
> Key: BIGTOP-852
> URL: https://issues.apache.org/jira/browse/BIGTOP-852
> Project: Bigtop
> Issue Type: Bug
> Components: Deployment
> Affects Versions: 0.6.0
> Reporter: Mark Grover
> Fix For: 0.6.0
>
>
> There are a few things that can be improved regarding the HDFS helper script introduced by BIGTOP-547. Some of these issues have been discussed in BIGTOP-637.
> 1. The script seems to create user directories for users "root" and "jenkins" but not the current user running the script. I think it will be a good idea to add in the script the commands to create the /user/$USER directory in HDFS. Of course, we should be careful in case the user running the command is root or jenkins, in that case a simple mkdir command would throw error given that those directories already exist.
> 2. The script uses sudo which requires login shell. However, login shells are not available for certain use cases like init scripts. Consequently, we should consider using su instead.
> 3. Look into how the helper script can be made faster
> Folks, please feel free to add/edit if there is something I missed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira