You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Roshan Punnoose <ro...@gmail.com> on 2013/05/09 19:22:48 UTC
Accumulo and Bigtop Hadoop
I have a feeling this has already been asked, but I could not find the
answer on google.
Bigtop is installing hadoop 2.0.2-alpha into /usr/lib/hadoop,
/usr/lib/hadoop-hdfs, /usr/lib/hadoop-yarn, etc. However, Accumulo is setup
to have the "general.classpaths" of accumulo-site to point
/usr/lib/hadoop/(share)/(hdfs|yarn)/*.jar.
As a quick workaround I had to edit the accumulo-env.sh and
accumulo-site.sh:
accumulo-env.sh
###for bigtop
export HADOOP_COMMON_DIR="./"
export HADOOP_COMMON_LIB_JARS_DIR="lib"
export HADOOP_COMMON_LIB_NATIVE_DIR="lib/native"
export HDFS_DIR="./"
export HDFS_LIB_JARS_DIR="lib"
export YARN_DIR="./"
export YARN_LIB_JARS_DIR="lib"
export MAPRED_DIR="./"
export MAPRED_LIB_JARS_DIR="lib"
export HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-"/usr/lib/hadoop/libexec"}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}
export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-"/usr/lib/hadoop"}
export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-"/usr/lib/hadoop-hdfs"}
export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/usr/lib/hadoop-mapreduce"}
export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/lib/hadoop-yarn"}
###for bigtop
accumulo-site.sh (added to the general.classpaths)
$HADOOP_MAPRED_HOME/.*.jar,
$HADOOP_MAPRED_HOME/$MAPRED_LIB_JARS_DIR/.*.jar,
$HADOOP_YARN_HOME/.*.jar,
$HADOOP_YARN_HOME/$YARN_LIB_JARS_DIR/.*.jar,
$HADOOP_HDFS_HOME/[^.].*.jar,
$HADOOP_HDFS_HOME/$HDFS_LIB_JARS_DIR/.*.jar,
This above is working. Has anyone ran into this and found a better solution?
Thanks!
Roshan
Re: Accumulo and Bigtop Hadoop
Posted by Christopher <ct...@apache.org>.
This is precisely why these classpaths are configurable. My personal
goal is that, in the future (1.6.0 and later), the RPMs/DEBs that we
provide will better support BigTop packages out of the box without the
need to edit the configuration.
(Also see John's response about hadoop 2 support.)
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Thu, May 9, 2013 at 1:22 PM, Roshan Punnoose <ro...@gmail.com> wrote:
> I have a feeling this has already been asked, but I could not find the
> answer on google.
>
> Bigtop is installing hadoop 2.0.2-alpha into /usr/lib/hadoop,
> /usr/lib/hadoop-hdfs, /usr/lib/hadoop-yarn, etc. However, Accumulo is setup
> to have the "general.classpaths" of accumulo-site to point
> /usr/lib/hadoop/(share)/(hdfs|yarn)/*.jar.
>
> As a quick workaround I had to edit the accumulo-env.sh and
> accumulo-site.sh:
>
> accumulo-env.sh
> ###for bigtop
> export HADOOP_COMMON_DIR="./"
> export HADOOP_COMMON_LIB_JARS_DIR="lib"
> export HADOOP_COMMON_LIB_NATIVE_DIR="lib/native"
> export HDFS_DIR="./"
> export HDFS_LIB_JARS_DIR="lib"
> export YARN_DIR="./"
> export YARN_LIB_JARS_DIR="lib"
> export MAPRED_DIR="./"
> export MAPRED_LIB_JARS_DIR="lib"
>
> export HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-"/usr/lib/hadoop/libexec"}
> export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}
> export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-"/usr/lib/hadoop"}
> export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-"/usr/lib/hadoop-hdfs"}
> export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/usr/lib/hadoop-mapreduce"}
> export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/lib/hadoop-yarn"}
> ###for bigtop
>
> accumulo-site.sh (added to the general.classpaths)
> $HADOOP_MAPRED_HOME/.*.jar,
> $HADOOP_MAPRED_HOME/$MAPRED_LIB_JARS_DIR/.*.jar,
> $HADOOP_YARN_HOME/.*.jar,
> $HADOOP_YARN_HOME/$YARN_LIB_JARS_DIR/.*.jar,
> $HADOOP_HDFS_HOME/[^.].*.jar,
> $HADOOP_HDFS_HOME/$HDFS_LIB_JARS_DIR/.*.jar,
>
> This above is working. Has anyone ran into this and found a better solution?
>
> Thanks!
>
> Roshan
Re: Accumulo and Bigtop Hadoop
Posted by Roshan Punnoose <ro...@gmail.com>.
Yeah I am using the accumulo 1.5 RC that Chris sent out a few days ago. We
are tied to using hadoop 2 currently. Just was not sure if someone had seen
this before.
On Thu, May 9, 2013 at 1:28 PM, John Vines <vi...@apache.org> wrote:
> Accumulo 1.4 does not work against hadoop 2 without changing the poms and
> recompiling. Accumulo 1.5, not yet released, will support both with
> recompilation. One option is to use Bigtop-0.3.x, which uses hadoop-1.0.1
> if I remember correctly.
>
>
> On Thu, May 9, 2013 at 1:22 PM, Roshan Punnoose <ro...@gmail.com> wrote:
>
>> I have a feeling this has already been asked, but I could not find the
>> answer on google.
>>
>> Bigtop is installing hadoop 2.0.2-alpha into /usr/lib/hadoop,
>> /usr/lib/hadoop-hdfs, /usr/lib/hadoop-yarn, etc. However, Accumulo is setup
>> to have the "general.classpaths" of accumulo-site to point
>> /usr/lib/hadoop/(share)/(hdfs|yarn)/*.jar.
>>
>> As a quick workaround I had to edit the accumulo-env.sh and
>> accumulo-site.sh:
>>
>> accumulo-env.sh
>> ###for bigtop
>> export HADOOP_COMMON_DIR="./"
>> export HADOOP_COMMON_LIB_JARS_DIR="lib"
>> export HADOOP_COMMON_LIB_NATIVE_DIR="lib/native"
>> export HDFS_DIR="./"
>> export HDFS_LIB_JARS_DIR="lib"
>> export YARN_DIR="./"
>> export YARN_LIB_JARS_DIR="lib"
>> export MAPRED_DIR="./"
>> export MAPRED_LIB_JARS_DIR="lib"
>>
>> export HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-"/usr/lib/hadoop/libexec"}
>> export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}
>> export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-"/usr/lib/hadoop"}
>> export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-"/usr/lib/hadoop-hdfs"}
>> export
>> HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/usr/lib/hadoop-mapreduce"}
>> export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/lib/hadoop-yarn"}
>> ###for bigtop
>>
>> accumulo-site.sh (added to the general.classpaths)
>> $HADOOP_MAPRED_HOME/.*.jar,
>> $HADOOP_MAPRED_HOME/$MAPRED_LIB_JARS_DIR/.*.jar,
>> $HADOOP_YARN_HOME/.*.jar,
>> $HADOOP_YARN_HOME/$YARN_LIB_JARS_DIR/.*.jar,
>> $HADOOP_HDFS_HOME/[^.].*.jar,
>> $HADOOP_HDFS_HOME/$HDFS_LIB_JARS_DIR/.*.jar,
>>
>> This above is working. Has anyone ran into this and found a better
>> solution?
>>
>> Thanks!
>>
>> Roshan
>>
>
>
Re: Accumulo and Bigtop Hadoop
Posted by John Vines <vi...@apache.org>.
Accumulo 1.4 does not work against hadoop 2 without changing the poms and
recompiling. Accumulo 1.5, not yet released, will support both with
recompilation. One option is to use Bigtop-0.3.x, which uses hadoop-1.0.1
if I remember correctly.
On Thu, May 9, 2013 at 1:22 PM, Roshan Punnoose <ro...@gmail.com> wrote:
> I have a feeling this has already been asked, but I could not find the
> answer on google.
>
> Bigtop is installing hadoop 2.0.2-alpha into /usr/lib/hadoop,
> /usr/lib/hadoop-hdfs, /usr/lib/hadoop-yarn, etc. However, Accumulo is setup
> to have the "general.classpaths" of accumulo-site to point
> /usr/lib/hadoop/(share)/(hdfs|yarn)/*.jar.
>
> As a quick workaround I had to edit the accumulo-env.sh and
> accumulo-site.sh:
>
> accumulo-env.sh
> ###for bigtop
> export HADOOP_COMMON_DIR="./"
> export HADOOP_COMMON_LIB_JARS_DIR="lib"
> export HADOOP_COMMON_LIB_NATIVE_DIR="lib/native"
> export HDFS_DIR="./"
> export HDFS_LIB_JARS_DIR="lib"
> export YARN_DIR="./"
> export YARN_LIB_JARS_DIR="lib"
> export MAPRED_DIR="./"
> export MAPRED_LIB_JARS_DIR="lib"
>
> export HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-"/usr/lib/hadoop/libexec"}
> export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop/conf"}
> export HADOOP_COMMON_HOME=${HADOOP_COMMON_HOME:-"/usr/lib/hadoop"}
> export HADOOP_HDFS_HOME=${HADOOP_HDFS_HOME:-"/usr/lib/hadoop-hdfs"}
> export
> HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-"/usr/lib/hadoop-mapreduce"}
> export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/lib/hadoop-yarn"}
> ###for bigtop
>
> accumulo-site.sh (added to the general.classpaths)
> $HADOOP_MAPRED_HOME/.*.jar,
> $HADOOP_MAPRED_HOME/$MAPRED_LIB_JARS_DIR/.*.jar,
> $HADOOP_YARN_HOME/.*.jar,
> $HADOOP_YARN_HOME/$YARN_LIB_JARS_DIR/.*.jar,
> $HADOOP_HDFS_HOME/[^.].*.jar,
> $HADOOP_HDFS_HOME/$HDFS_LIB_JARS_DIR/.*.jar,
>
> This above is working. Has anyone ran into this and found a better
> solution?
>
> Thanks!
>
> Roshan
>