Re: chukwa download

[Ken White <ke...@mailcatch.com>]

>Your JAVA_HOME is pointing to a java 1.4 JDK; needs to be 1.6.

Thank you!!! :)
Indeed - JAVA_HOME was not set so it probably used some old Java engine I had 
lying around. 

I would appreciate some additional help in setting this up, I hope I'm not 
bothering you too much. 

I am following this guide:
http://wiki.apache.org/hadoop/Chukwa_Quick_Start

Right now I am trying to run `bash bin/jettyCollector.sh` but I get this 
exception:
java.lang.ClassNotFoundException: 
org.apache.hadoop.chukwa.datacollection.collector.CollectorStub

I guess my classpath is incomplete, but I can't figure out what is missing (in 
which jar CollectorStub is). From jettyCollector.sh:
$ java ... -classpath ${CHUKWA_CONF_DIR}:${HADOOP_CONF_DIR}:${CLASSPATH}:
${CHUKWA_CORE}:${COMMON}:${HADOOP_JAR} ...

Also, there are some things that are not clear to me in conf/chukwa-collector-
conf.xml:

1) writer.hdfs.filesystem - I want to output to HDFS. Does that mean I have to 
specify hdfs://... style path? In Quick Start is says "path to the namenode", 
but that is probably not enough?

2) chukwaCollector.outputDir - is this the subdir of previous setting or is 
this the local output directory that Collector writes to ebfore it commits to 
HDFS?

3) chukwaCollector.rotateInterval - the default value is 5 minutes, is there 
any reason for it to be specified in milliseconds? Is this the time between 
commits to HDFS?

Thank you again for your help, I appreciate it.

Kind regards,

Ken

Re: chukwa download

[Ariel Rabkin <as...@gmail.com>]

Hrm.
 That document is badly out of date. You should probably be looking at
http://hadoop.apache.org/chukwa/docs/current/admin.html#2.+Configure+the+Agent
instead.

On Wed, Jul 1, 2009 at 5:25 AM, Ken White<ke...@mailcatch.com> wrote:
>
>>Your JAVA_HOME is pointing to a java 1.4 JDK; needs to be 1.6.
> I am following this guide:
> http://wiki.apache.org/hadoop/Chukwa_Quick_Start
>
> Right now I am trying to run `bash bin/jettyCollector.sh` but I get this
> exception:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.chukwa.datacollection.collector.CollectorStub
>
> I guess my classpath is incomplete, but I can't figure out what is missing (in
> which jar CollectorStub is). From jettyCollector.sh:
> $ java ... -classpath ${CHUKWA_CONF_DIR}:${HADOOP_CONF_DIR}:${CLASSPATH}:
> ${CHUKWA_CORE}:${COMMON}:${HADOOP_JAR} ...

You need to move the jars from build to the chukwa root directory.

> Also, there are some things that are not clear to me in conf/chukwa-collector-
> conf.xml:
>
> 1) writer.hdfs.filesystem - I want to output to HDFS. Does that mean I have to
> specify hdfs://... style path? In Quick Start is says "path to the namenode",
> but that is probably not enough?

Actually, that's no longer needed, so long as you define
HADOOP_CONF_DIR in chukwa-env.sh

> 2) chukwaCollector.outputDir - is this the subdir of previous setting or is
> this the local output directory that Collector writes to ebfore it commits to
> HDFS?

I believe this is the directory on HDFS. But I think the default
setting is fine.

> 3) chukwaCollector.rotateInterval - the default value is 5 minutes, is there
> any reason for it to be specified in milliseconds? Is this the time between
> commits to HDFS?

This upper-bounds the time between commits to HDFS; it may happen more
frequently at high load. The issue is that HDFS commits when a file
closes, or else at the end of a block. This specifies when to close
the file.  Specified in milliseconds because we wanted to use millis
thoughought to avoid unit confusion.

--Ari

-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department