You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by Kirk True <ki...@mustardgrain.com> on 2010/04/28 18:37:09 UTC
Chukwa can't find Demux class
Hi guys,
I have a custom Demux that I need to run to process my input, but I'm
getting ClassNotFoundException when running in Hadoop. This is with the
released 0.4.0 build.
I've done the following:
1. I put my Demux class in the correct package
(org.apache.hadoop.chukwa.extraction.demux.processor.mapper)
2. I've added the JAR containing the Demux implementation to HDFS at
/chuka/demux
3. I've added an alias to it in chukwa-demux-conf.xml
The map/reduce job is picking up on the fact that I have a custom Demux
and is trying to load it, but I get a ClassNotFoundException. The
HDFS-based URL to the JAR is showing up in the job configuration in
Hadoop, which is another evidence that Chukwa and Hadoop know where the
JAR lives and that it's part of the Chukwa-initiated job.
My Demux is very simple. I've stripped it down to a System.out.println
with dependencies on no other classes/JARs other than Chukwa, Hadoop,
and the core JDK. I've double-checked that my JAR is being built up
correctly. I'm completely flummoxed as to what I'm doing wrong.
Any ideas what I'm missing? What other information can I provide?
Thanks!
Kirk
Re: Chukwa can't find Demux class
Posted by Kirk True <ki...@mustardgrain.com>.
Hi Eric,
I updated MapProcessorFactory.getProcessor to dump the URLs from
the URLClassLoader from the MapProcessorFactory.class. This is
what I see:
file:/home/kirk/bin/hadoop-0.20.2/conf/
file:/home/kirk/bin/jdk1.6.0_18/lib/tools.jar
file:/home/kirk/bin/hadoop-0.20.2/
file:/home/kirk/bin/hadoop-0.20.2/hadoop-0.20.2-core.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-cli-1.2.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-codec-1.3.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-el-1.0.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-httpclient-3.0.1.ja
r
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-logging-1.0.4.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-logging-api-1.0.4.j
ar
file:/home/kirk/bin/hadoop-0.20.2/lib/commons-net-1.4.1.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/core-3.1.1.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jets3t-0.6.1.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jetty-6.1.14.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jetty-util-6.1.14.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/junit-3.8.1.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/kfs-0.2.2.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/log4j-1.2.15.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/mockito-all-1.8.0.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/oro-2.0.8.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/xmlenc-0.52.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar
file:/home/kirk/bin/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar
file:/tmp/hadoop-kirk/mapred/local/taskTracker/jobcache/job_20100
4281320_0001/attempt_201004281320_0001_m_000000_0/work/
file:/tmp/hadoop-kirk/mapred/local/taskTracker/jobcache/job_20100
4281320_0001/jars/classes
file:/tmp/hadoop-kirk/mapred/local/taskTracker/jobcache/job_20100
4281320_0001/jars/
file:/tmp/hadoop-kirk/mapred/local/taskTracker/jobcache/job_20100
4281320_0001/attempt_201004281320_0001_m_000000_0/work/
Is that the expected classpath? I don't see any reference to my
JAR or the Chukwa JARs.
Also, when I try to view the contents of my
"job_<timestamp>_0001" directory, it's automatically removed, so
I can't really do any forensics after the fact. I know this is
probably a Hadoop question, is it possible to prevent that
auto-removal from occurring?
Thanks,
Kirk
On Wed, 28 Apr 2010 13:16 -0700, "Kirk True"
<ki...@mustardgrain.com> wrote:
> Hi Eric,
>
> On 4/28/10 10:23 AM, Eric Yang wrote:
> > Hi Kirk,
> >
> > Is the ownership of the jar file setup correctly as the user
that runs
> > demux?
>
> When browsing via the NameNode web UI, it lists permissions of
> "rw-r--r--" and "kirk" as the owner (which is also the user ID
running
> the Hadoop and Chukwa processes).
>
> > You may find more information by looking at running mapper
task or
> > reducer task, and try to find out the task attempt shell
script.
>
> Where is the task attempt shell script located?
>
> > Make sure
> > the files are downloaded correctly from distributed cache,
and referenced in
> > the locally generated jar file. Hope this helps.
> >
>
> Sorry for asking such basic questions, but where is the locally
> generated JAR file found? I'm assuming under
/tmp/hadoop-<user>, by
> default? I saw one file named job_<timstamp>.jar but it
appeared to be a
> byte-for-byte copy of chukwa-core-0.4.0.jar, i.e. my
"XmlBasedDemux"
> class was nowhere to be found.
>
> Thanks,
> Kirk
>
> > Regards,
> > Eric
> >
> > On 4/28/10 9:37 AM, "Kirk True"<ki...@mustardgrain.com> wrote:
> >
> >
> >> Hi guys,
> >>
> >> I have a custom Demux that I need to run to process my
input, but I'm getting
> >> ClassNotFoundException when running in Hadoop. This is with
the released 0.4.0
> >> build.
> >>
> >> I've done the following:
> >>
> >> 1. I put my Demux class in the correct package
> >> (org.apache.hadoop.chukwa.extraction.demux.processor.mapper)
> >> 2. I've added the JAR containing the Demux implementation to
HDFS at
> >> /chuka/demux
> >> 3. I've added an alias to it in chukwa-demux-conf.xml
> >>
> >> The map/reduce job is picking up on the fact that I have a
custom Demux and is
> >> trying to load it, but I get a ClassNotFoundException. The
HDFS-based URL to
> >> the JAR is showing up in the job configuration in Hadoop,
which is another
> >> evidence that Chukwa and Hadoop know where the JAR lives and
that it's part of
> >> the Chukwa-initiated job.
> >>
> >> My Demux is very simple. I've stripped it down to a
System.out.println with
> >> dependencies on no other classes/JARs other than Chukwa,
Hadoop, and the core
> >> JDK. I've double-checked that my JAR is being built up
correctly. I'm
> >> completely flummoxed as to what I'm doing wrong.
> >>
> >> Any ideas what I'm missing? What other information can I
provide?
> >>
> >> Thanks!
> >> Kirk
> >>
> >>
> >
>
Re: Chukwa can't find Demux class
Posted by Kirk True <ki...@mustardgrain.com>.
Hi Eric,
On 4/28/10 10:23 AM, Eric Yang wrote:
> Hi Kirk,
>
> Is the ownership of the jar file setup correctly as the user that runs
> demux?
When browsing via the NameNode web UI, it lists permissions of
"rw-r--r--" and "kirk" as the owner (which is also the user ID running
the Hadoop and Chukwa processes).
> You may find more information by looking at running mapper task or
> reducer task, and try to find out the task attempt shell script.
Where is the task attempt shell script located?
> Make sure
> the files are downloaded correctly from distributed cache, and referenced in
> the locally generated jar file. Hope this helps.
>
Sorry for asking such basic questions, but where is the locally
generated JAR file found? I'm assuming under /tmp/hadoop-<user>, by
default? I saw one file named job_<timstamp>.jar but it appeared to be a
byte-for-byte copy of chukwa-core-0.4.0.jar, i.e. my "XmlBasedDemux"
class was nowhere to be found.
Thanks,
Kirk
> Regards,
> Eric
>
> On 4/28/10 9:37 AM, "Kirk True"<ki...@mustardgrain.com> wrote:
>
>
>> Hi guys,
>>
>> I have a custom Demux that I need to run to process my input, but I'm getting
>> ClassNotFoundException when running in Hadoop. This is with the released 0.4.0
>> build.
>>
>> I've done the following:
>>
>> 1. I put my Demux class in the correct package
>> (org.apache.hadoop.chukwa.extraction.demux.processor.mapper)
>> 2. I've added the JAR containing the Demux implementation to HDFS at
>> /chuka/demux
>> 3. I've added an alias to it in chukwa-demux-conf.xml
>>
>> The map/reduce job is picking up on the fact that I have a custom Demux and is
>> trying to load it, but I get a ClassNotFoundException. The HDFS-based URL to
>> the JAR is showing up in the job configuration in Hadoop, which is another
>> evidence that Chukwa and Hadoop know where the JAR lives and that it's part of
>> the Chukwa-initiated job.
>>
>> My Demux is very simple. I've stripped it down to a System.out.println with
>> dependencies on no other classes/JARs other than Chukwa, Hadoop, and the core
>> JDK. I've double-checked that my JAR is being built up correctly. I'm
>> completely flummoxed as to what I'm doing wrong.
>>
>> Any ideas what I'm missing? What other information can I provide?
>>
>> Thanks!
>> Kirk
>>
>>
>
Re: Chukwa can't find Demux class
Posted by Eric Yang <ey...@yahoo-inc.com>.
Hi Kirk,
Is the ownership of the jar file setup correctly as the user that runs
demux? You may find more information by looking at running mapper task or
reducer task, and try to find out the task attempt shell script. Make sure
the files are downloaded correctly from distributed cache, and referenced in
the locally generated jar file. Hope this helps.
Regards,
Eric
On 4/28/10 9:37 AM, "Kirk True" <ki...@mustardgrain.com> wrote:
> Hi guys,
>
> I have a custom Demux that I need to run to process my input, but I'm getting
> ClassNotFoundException when running in Hadoop. This is with the released 0.4.0
> build.
>
> I've done the following:
>
> 1. I put my Demux class in the correct package
> (org.apache.hadoop.chukwa.extraction.demux.processor.mapper)
> 2. I've added the JAR containing the Demux implementation to HDFS at
> /chuka/demux
> 3. I've added an alias to it in chukwa-demux-conf.xml
>
> The map/reduce job is picking up on the fact that I have a custom Demux and is
> trying to load it, but I get a ClassNotFoundException. The HDFS-based URL to
> the JAR is showing up in the job configuration in Hadoop, which is another
> evidence that Chukwa and Hadoop know where the JAR lives and that it's part of
> the Chukwa-initiated job.
>
> My Demux is very simple. I've stripped it down to a System.out.println with
> dependencies on no other classes/JARs other than Chukwa, Hadoop, and the core
> JDK. I've double-checked that my JAR is being built up correctly. I'm
> completely flummoxed as to what I'm doing wrong.
>
> Any ideas what I'm missing? What other information can I provide?
>
> Thanks!
> Kirk
>