You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by Ariel Rabkin <as...@gmail.com> on 2009/04/29 07:34:47 UTC
which hadoop jars to use?
Hi all.
So it looks like we're now hardcoding which hadoop jar to use in
chukwa-config.sh -- we always use the ones in hadoopjars in preference
to trying to autodetect, and we hardcode the versions we want. Is this
a stopgap, or a permanent measure?
--Ari
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: which hadoop jars to use?
Posted by Ariel Rabkin <as...@gmail.com>.
This isn't entirely true.
If you look at agent.sh, it uses HADOOP_20_JAR, only. Which is
hardcoded in bin/chukwa-config.sh.
That script also sets HADOOP_18_JAR, which as near as I can tell is
used nowhere.
Also. Can we document some of our Hadoop version discoveries? It's
not at all obvious from the current admin guide that the processing
cluster needs to be running 0.18, and that the cluster being monitored
ought to be 0.20.
--Ari
On Wed, Apr 29, 2009 at 10:19 AM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Hi Ari,
>
> Hadoop jars should be autodetected from either HADOOP_HOME, or hadoopjars
> directory. For the agent side, you could remove the hadoop-0.18.3.jar from
> the hadoopjars directory to make data collection work with new version of
> hadoop. The current stopgap is messy because demux depends on hadoop-0.18.3
> jar file. It was not the best approach to place hadoop-0.18.3 jar file in
> hadoopjars directory.
>
> Short term, use HADOOP_HOME to control location of HADOOP jar file.
> Mid term, create individual RPM file which bundles the right hadoop jars.
> Long term, Upgrade demux pipeline to use hadoop 0.20.0+.
>
> Regards,
> Eric
>
> On 4/28/09 10:34 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
>
>> Hi all.
>>
>> So it looks like we're now hardcoding which hadoop jar to use in
>> chukwa-config.sh -- we always use the ones in hadoopjars in preference
>> to trying to autodetect, and we hardcode the versions we want. Is this
>> a stopgap, or a permanent measure?
>>
>> --Ari
>
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: which hadoop jars to use?
Posted by Eric Yang <ey...@yahoo-inc.com>.
Hi Ari,
Hadoop jars should be autodetected from either HADOOP_HOME, or hadoopjars
directory. For the agent side, you could remove the hadoop-0.18.3.jar from
the hadoopjars directory to make data collection work with new version of
hadoop. The current stopgap is messy because demux depends on hadoop-0.18.3
jar file. It was not the best approach to place hadoop-0.18.3 jar file in
hadoopjars directory.
Short term, use HADOOP_HOME to control location of HADOOP jar file.
Mid term, create individual RPM file which bundles the right hadoop jars.
Long term, Upgrade demux pipeline to use hadoop 0.20.0+.
Regards,
Eric
On 4/28/09 10:34 PM, "Ariel Rabkin" <as...@gmail.com> wrote:
> Hi all.
>
> So it looks like we're now hardcoding which hadoop jar to use in
> chukwa-config.sh -- we always use the ones in hadoopjars in preference
> to trying to autodetect, and we hardcode the versions we want. Is this
> a stopgap, or a permanent measure?
>
> --Ari