You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by C G <pa...@yahoo.com> on 2007/10/12 23:43:02 UTC
Question about valueaggregators in 0.14.1...
Hi All:
I am tearing out my hair trying to get a simple valueaggregator to run. This seems like an
easy enough thing, but I am consistently getting hit with ClassNotFoundException which
makes no sense to me. Any help would be immensely appreciated. Note that I've had
valueaggregators running on 0.13.*. I'm a little concerned there is a bug in 0.14.1. On the
other hand, maybe I'm just blind to something obvious and just need a boot to the head...
The environment is hadoop-0.14.1. I am running Java 1.6:
$ java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
The class name is called testAggregator, and exists in my jar file:
$ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
1826 Fri Oct 12 16:30:14 EDT 2007 com/company/hadoop/metrics/testAggregator.class
My XML file is testAggregator.xml and looks like:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>aggregator.descriptor.num</name>
<value>1</value>
</property>
<property>
<name>aggregator.descriptor.0</name>
<value>UserDefined,com.company.hadoop.metrics.testAggregator</value>
</property>
</configuration>
My code couldn't be simpler:
package com.company.hadoop.metrics;
// snip a bunch of imports
public class testAggregator extends ValueAggregatorBaseDescriptor {
public ArrayList<Entry> generateKeyValPairs(Object key, Object val) {
String fields[] = val.toString().split("\t");
ArrayList<Entry> retv = new ArrayList<Entry>();
retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text( fields[ 2 ])));
return retv;
}
public void configure(JobConf job) {}
}
I invoke things this way:
bin/hadoop jar hadoop-0.14.1-dev-company.jar org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1 /output/set1 8 textinputformat testAggregator.xml
All my jobs fail with a ClassNotFoundException like the following traceback:
2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 8
2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.company.hadoop.metrics.testAggregator
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance
(UserDefinedValueAggregatorDescriptor.java:56)
at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
at org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.<init>(UserDefinedValueAggregatorDescriptor.java:75)
at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggregatorDescriptor(ValueAggregatorJobBase.java:49)
at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregatorDescriptors(ValueAggregatorJobBase.java:60)
at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMySpec(ValueAggregatorJobBase.java:69)
at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(ValueAggregatorJobBase.java:37)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
Caused by: java.lang.ClassNotFoundException: com.company.hadoop.metrics.testAggregator
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance
(UserDefinedValueAggregatorDescriptor.java:51)
... 13 more
As near as I can tell I am doing everything correctly. Other code existing in this jar file is
running just fine - in fact it was used to produce the input to this test. I've checked my
systems to make I've got the jar file (and XML file) onto each of the 4 compute nodes in the
system.
Can anybody spot my problem or tell me what to do to get this going?
Thanks,
C G
---------------------------------
Tonight's top picks. What will you watch tonight? Preview the hottest shows on Yahoo! TV.
RE: Question about valueaggregators in 0.14.1...
Posted by Runping Qi <ru...@yahoo-inc.com>.
Glad it works.
HADOOP-1622 should fix the problem properly.
Until then, the users have to use this kind of hacky obscure workaround:)
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 8:48 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Question about valueaggregators in 0.14.1...
>
> Hi Runping and All:
>
> That fixed the problem. Of course my aggregator is now failing for a
> different reason, but that's an error in my code that I can fix.
>
> I am extremely grateful for your assistance!
>
> Thanks,
> C G
>
> Runping Qi <ru...@yahoo-inc.com> wrote:
>
>
> I am sorry I overlooked something.
> Try to add the following:
>
> job.setJarByClass(MyValueAggregatorJob.class);
> JobClient.runJob(job);
>
> To your main.
>
> Runping
>
> > -----Original Message-----
> > From: C G [mailto:parallelguy@yahoo.com]
> > Sent: Friday, October 12, 2007 8:05 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: RE: Question about valueaggregators in 0.14.1...
> >
> > Hi Runping and All:
> >
> > I took your suggestion, renamed my original code to testAggregatorPlugin
> > and wrote a class testAggregator which contained the main as you show
> > here. It also fails with the same traceback...so it seems something is
> > profoundly wrong here.
> >
> > Is anybody else using the aggregation classes in 0.14.1 with success?
> >
> > This is ironic and frustrating...I have several other programs which use
> > map/reduce that are quite complex and those are running just fine. This
> > little plugin is all that stands in the way of delivering the system :-
> (.
> >
> > Any other comments/help, etc. most welcome...
> >
> > Thanks,
> > C G
> >
> > Runping Qi wrote:
> >
> > Hadoop ignored the jar file you specified.
> > This is a known problem andhopefully HADOOP-1622
> > (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the
> problem.
> >
> > In the meantime, you can create a class like below and add it to your
> jar:
> >
> > package your_package;
> >
> >
> > public class MyValueAggregatorJob{
> >
> > public static void main(String args[]) throws IOException {
> > JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> > JobClient.runJob(job);
> > }
> > }
> >
> >
> > And use the above class as the main class when you submit hadoop job:
> > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > your_package.MyValueAggregatorJob /input/set1 /output/set1 8
> > textinputformat
> > testAggregator.xml
> >
> >
> > Hope this solve your problem.
> >
> > Runping
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: C G [mailto:parallelguy@yahoo.com]
> > > Sent: Friday, October 12, 2007 2:43 PM
> > > To: hadoop-user@lucene.apache.org
> > > Subject: Question about valueaggregators in 0.14.1...
> > >
> > > Hi All:
> > >
> > > I am tearing out my hair trying to get a simple valueaggregator to
> run.
> > > This seems like an
> > > easy enough thing, but I am consistently getting hit with
> > > ClassNotFoundException which
> > > makes no sense to me. Any help would be immensely appreciated. Note
> > > that I've had
> > > valueaggregators running on 0.13.*. I'm a little concerned there is a
> > bug
> > > in 0.14.1. On the
> > > other hand, maybe I'm just blind to something obvious and just need a
> > > boot to the head...
> > >
> > > The environment is hadoop-0.14.1. I am running Java 1.6:
> > > $ java -version
> > > java version "1.6.0_02"
> > > Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> > > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> > >
> > > The class name is called testAggregator, and exists in my jar file:
> > > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> > > 1826 Fri Oct 12 16:30:14 EDT 2007
> > > com/company/hadoop/metrics/testAggregator.class
> > >
> > > My XML file is testAggregator.xml and looks like:
> > >
> > >
> > >
> > >
> >
> > > aggregator.descriptor.num
> > > 1
> > >
> >
> > >
> >
> > > aggregator.descriptor.0
> > > UserDefined,com.company.hadoop.metrics.testAggregator
> > >
> >
> > >
> > >
> > > My code couldn't be simpler:
> > >
> > > package com.company.hadoop.metrics;
> > > // snip a bunch of imports
> > > public class testAggregator extends ValueAggregatorBaseDescriptor {
> > > public ArrayList generateKeyValPairs(Object key, Object val) {
> > > String fields[] = val.toString().split("\t");
> > > ArrayList retv = new ArrayList();
> > > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> > > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> > > fields[ 2 ])));
> > > return retv;
> > > }
> > > public void configure(JobConf job) {}
> > > }
> > >
> > > I invoke things this way:
> > >
> > > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> > > /output/set1 8 textinputformat testAggregator.xml
> > >
> > > All my jobs fail with a ClassNotFoundException like the following
> > > traceback:
> > >
> > > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > > Initializing JVM Metrics with processName=MAP, sessionId=
> > > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> > > numReduceTasks: 8
> > > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker:
> Error
> > > running child
> > > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > > com.company.hadoop.metrics.testAggregator
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createInstance
> > > (UserDefinedValueAggregatorDescriptor.java:56)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.(UserDefinedValueAggregatorDescriptor.java:75)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> > > egatorDescriptor(ValueAggregatorJobBase.java:49)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> > > rDescriptors(ValueAggregatorJobBase.java:60)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> > > Spec(ValueAggregatorJobBase.java:69)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> > > lueAggregatorJobBase.java:37)
> > > at
> > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > at
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > > )
> > > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> > > at
> > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > at
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > > )
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> > > at
> > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> > > Caused by: java.lang.ClassNotFoundException:
> > > com.company.hadoop.metrics.testAggregator
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > > at java.lang.Class.forName0(Native Method)
> > > at java.lang.Class.forName(Class.java:247)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createInstance
> > > (UserDefinedValueAggregatorDescriptor.java:51)
> > > ... 13 more
> > >
> > > As near as I can tell I am doing everything correctly. Other code
> > > existing in this jar file is
> > > running just fine - in fact it was used to produce the input to this
> > > test. I've checked my
> > > systems to make I've got the jar file (and XML file) onto each of the
> 4
> > > compute nodes in the
> > > system.
> > >
> > > Can anybody spot my problem or tell me what to do to get this going?
> > >
> > > Thanks,
> > > C G
> > >
> > >
> > > ---------------------------------
> > > Tonight's top picks. What will you watch tonight? Preview the hottest
> > > shows on Yahoo! TV.
> >
> >
> >
> >
> > ---------------------------------
> > Be a better Heartthrob. Get better relationship answers from someone who
> > knows.
> > Yahoo! Answers - Check it out.
>
>
>
>
> ---------------------------------
> Catch up on fall's hot new shows on Yahoo! TV. Watch previews, get
> listings, and more!
RE: Question about valueaggregators in 0.14.1...
Posted by C G <pa...@yahoo.com>.
Hi Runping and All:
That fixed the problem. Of course my aggregator is now failing for a different reason, but that's an error in my code that I can fix.
I am extremely grateful for your assistance!
Thanks,
C G
Runping Qi <ru...@yahoo-inc.com> wrote:
I am sorry I overlooked something.
Try to add the following:
job.setJarByClass(MyValueAggregatorJob.class);
JobClient.runJob(job);
To your main.
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 8:05 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Question about valueaggregators in 0.14.1...
>
> Hi Runping and All:
>
> I took your suggestion, renamed my original code to testAggregatorPlugin
> and wrote a class testAggregator which contained the main as you show
> here. It also fails with the same traceback...so it seems something is
> profoundly wrong here.
>
> Is anybody else using the aggregation classes in 0.14.1 with success?
>
> This is ironic and frustrating...I have several other programs which use
> map/reduce that are quite complex and those are running just fine. This
> little plugin is all that stands in the way of delivering the system :-(.
>
> Any other comments/help, etc. most welcome...
>
> Thanks,
> C G
>
> Runping Qi wrote:
>
> Hadoop ignored the jar file you specified.
> This is a known problem andhopefully HADOOP-1622
> (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
>
> In the meantime, you can create a class like below and add it to your jar:
>
> package your_package;
>
>
> public class MyValueAggregatorJob{
>
> public static void main(String args[]) throws IOException {
> JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> JobClient.runJob(job);
> }
> }
>
>
> And use the above class as the main class when you submit hadoop job:
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> your_package.MyValueAggregatorJob /input/set1 /output/set1 8
> textinputformat
> testAggregator.xml
>
>
> Hope this solve your problem.
>
> Runping
>
>
>
>
> > -----Original Message-----
> > From: C G [mailto:parallelguy@yahoo.com]
> > Sent: Friday, October 12, 2007 2:43 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: Question about valueaggregators in 0.14.1...
> >
> > Hi All:
> >
> > I am tearing out my hair trying to get a simple valueaggregator to run.
> > This seems like an
> > easy enough thing, but I am consistently getting hit with
> > ClassNotFoundException which
> > makes no sense to me. Any help would be immensely appreciated. Note
> > that I've had
> > valueaggregators running on 0.13.*. I'm a little concerned there is a
> bug
> > in 0.14.1. On the
> > other hand, maybe I'm just blind to something obvious and just need a
> > boot to the head...
> >
> > The environment is hadoop-0.14.1. I am running Java 1.6:
> > $ java -version
> > java version "1.6.0_02"
> > Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> >
> > The class name is called testAggregator, and exists in my jar file:
> > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> > 1826 Fri Oct 12 16:30:14 EDT 2007
> > com/company/hadoop/metrics/testAggregator.class
> >
> > My XML file is testAggregator.xml and looks like:
> >
> >
> >
> >
>
> > aggregator.descriptor.num
> > 1
> >
>
> >
>
> > aggregator.descriptor.0
> > UserDefined,com.company.hadoop.metrics.testAggregator
> >
>
> >
> >
> > My code couldn't be simpler:
> >
> > package com.company.hadoop.metrics;
> > // snip a bunch of imports
> > public class testAggregator extends ValueAggregatorBaseDescriptor {
> > public ArrayList generateKeyValPairs(Object key, Object val) {
> > String fields[] = val.toString().split("\t");
> > ArrayList retv = new ArrayList();
> > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> > fields[ 2 ])));
> > return retv;
> > }
> > public void configure(JobConf job) {}
> > }
> >
> > I invoke things this way:
> >
> > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> > /output/set1 8 textinputformat testAggregator.xml
> >
> > All my jobs fail with a ClassNotFoundException like the following
> > traceback:
> >
> > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId=
> > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> > numReduceTasks: 8
> > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> > running child
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:56)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.(UserDefinedValueAggregatorDescriptor.java:75)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> > egatorDescriptor(ValueAggregatorJobBase.java:49)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> > rDescriptors(ValueAggregatorJobBase.java:60)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> > Spec(ValueAggregatorJobBase.java:69)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> > lueAggregatorJobBase.java:37)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> > at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> > Caused by: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:51)
> > ... 13 more
> >
> > As near as I can tell I am doing everything correctly. Other code
> > existing in this jar file is
> > running just fine - in fact it was used to produce the input to this
> > test. I've checked my
> > systems to make I've got the jar file (and XML file) onto each of the 4
> > compute nodes in the
> > system.
> >
> > Can anybody spot my problem or tell me what to do to get this going?
> >
> > Thanks,
> > C G
> >
> >
> > ---------------------------------
> > Tonight's top picks. What will you watch tonight? Preview the hottest
> > shows on Yahoo! TV.
>
>
>
>
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who
> knows.
> Yahoo! Answers - Check it out.
---------------------------------
Catch up on fall's hot new shows on Yahoo! TV. Watch previews, get listings, and more!
RE: Question about valueaggregators in 0.14.1...
Posted by Runping Qi <ru...@yahoo-inc.com>.
I am sorry I overlooked something.
Try to add the following:
job.setJarByClass(MyValueAggregatorJob.class);
JobClient.runJob(job);
To your main.
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 8:05 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Question about valueaggregators in 0.14.1...
>
> Hi Runping and All:
>
> I took your suggestion, renamed my original code to testAggregatorPlugin
> and wrote a class testAggregator which contained the main as you show
> here. It also fails with the same traceback...so it seems something is
> profoundly wrong here.
>
> Is anybody else using the aggregation classes in 0.14.1 with success?
>
> This is ironic and frustrating...I have several other programs which use
> map/reduce that are quite complex and those are running just fine. This
> little plugin is all that stands in the way of delivering the system :-(.
>
> Any other comments/help, etc. most welcome...
>
> Thanks,
> C G
>
> Runping Qi <ru...@yahoo-inc.com> wrote:
>
> Hadoop ignored the jar file you specified.
> This is a known problem andhopefully HADOOP-1622
> (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
>
> In the meantime, you can create a class like below and add it to your jar:
>
> package your_package;
>
>
> public class MyValueAggregatorJob{
>
> public static void main(String args[]) throws IOException {
> JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> JobClient.runJob(job);
> }
> }
>
>
> And use the above class as the main class when you submit hadoop job:
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> your_package.MyValueAggregatorJob /input/set1 /output/set1 8
> textinputformat
> testAggregator.xml
>
>
> Hope this solve your problem.
>
> Runping
>
>
>
>
> > -----Original Message-----
> > From: C G [mailto:parallelguy@yahoo.com]
> > Sent: Friday, October 12, 2007 2:43 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: Question about valueaggregators in 0.14.1...
> >
> > Hi All:
> >
> > I am tearing out my hair trying to get a simple valueaggregator to run.
> > This seems like an
> > easy enough thing, but I am consistently getting hit with
> > ClassNotFoundException which
> > makes no sense to me. Any help would be immensely appreciated. Note
> > that I've had
> > valueaggregators running on 0.13.*. I'm a little concerned there is a
> bug
> > in 0.14.1. On the
> > other hand, maybe I'm just blind to something obvious and just need a
> > boot to the head...
> >
> > The environment is hadoop-0.14.1. I am running Java 1.6:
> > $ java -version
> > java version "1.6.0_02"
> > Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> >
> > The class name is called testAggregator, and exists in my jar file:
> > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> > 1826 Fri Oct 12 16:30:14 EDT 2007
> > com/company/hadoop/metrics/testAggregator.class
> >
> > My XML file is testAggregator.xml and looks like:
> >
> >
> >
> >
>
> > aggregator.descriptor.num
> > 1
> >
>
> >
>
> > aggregator.descriptor.0
> > UserDefined,com.company.hadoop.metrics.testAggregator
> >
>
> >
> >
> > My code couldn't be simpler:
> >
> > package com.company.hadoop.metrics;
> > // snip a bunch of imports
> > public class testAggregator extends ValueAggregatorBaseDescriptor {
> > public ArrayList generateKeyValPairs(Object key, Object val) {
> > String fields[] = val.toString().split("\t");
> > ArrayList retv = new ArrayList();
> > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> > fields[ 2 ])));
> > return retv;
> > }
> > public void configure(JobConf job) {}
> > }
> >
> > I invoke things this way:
> >
> > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> > /output/set1 8 textinputformat testAggregator.xml
> >
> > All my jobs fail with a ClassNotFoundException like the following
> > traceback:
> >
> > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId=
> > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> > numReduceTasks: 8
> > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> > running child
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:56)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.(UserDefinedValueAggregatorDescriptor.java:75)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> > egatorDescriptor(ValueAggregatorJobBase.java:49)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> > rDescriptors(ValueAggregatorJobBase.java:60)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> > Spec(ValueAggregatorJobBase.java:69)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> > lueAggregatorJobBase.java:37)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> > at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> > Caused by: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:51)
> > ... 13 more
> >
> > As near as I can tell I am doing everything correctly. Other code
> > existing in this jar file is
> > running just fine - in fact it was used to produce the input to this
> > test. I've checked my
> > systems to make I've got the jar file (and XML file) onto each of the 4
> > compute nodes in the
> > system.
> >
> > Can anybody spot my problem or tell me what to do to get this going?
> >
> > Thanks,
> > C G
> >
> >
> > ---------------------------------
> > Tonight's top picks. What will you watch tonight? Preview the hottest
> > shows on Yahoo! TV.
>
>
>
>
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who
> knows.
> Yahoo! Answers - Check it out.
Re: Question about valueaggregators in 0.14.1...
Posted by Dennis Kubes <ku...@apache.org>.
HADOOP-1622 has a new patch available. That may solve your problem.
Dennis Kubes
C G wrote:
> Hi Runping and All:
>
> I took your suggestion, renamed my original code to testAggregatorPlugin and wrote a class testAggregator which contained the main as you show here. It also fails with the same traceback...so it seems something is profoundly wrong here.
>
> Is anybody else using the aggregation classes in 0.14.1 with success?
>
> This is ironic and frustrating...I have several other programs which use map/reduce that are quite complex and those are running just fine. This little plugin is all that stands in the way of delivering the system :-(.
>
> Any other comments/help, etc. most welcome...
>
> Thanks,
> C G
>
> Runping Qi <ru...@yahoo-inc.com> wrote:
>
> Hadoop ignored the jar file you specified.
> This is a known problem andhopefully HADOOP-1622
> (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
>
> In the meantime, you can create a class like below and add it to your jar:
>
> package your_package;
>
>
> public class MyValueAggregatorJob{
>
> public static void main(String args[]) throws IOException {
> JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> JobClient.runJob(job);
> }
> }
>
>
> And use the above class as the main class when you submit hadoop job:
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
> testAggregator.xml
>
>
> Hope this solve your problem.
>
> Runping
>
>
>
>
>> -----Original Message-----
>> From: C G [mailto:parallelguy@yahoo.com]
>> Sent: Friday, October 12, 2007 2:43 PM
>> To: hadoop-user@lucene.apache.org
>> Subject: Question about valueaggregators in 0.14.1...
>>
>> Hi All:
>>
>> I am tearing out my hair trying to get a simple valueaggregator to run.
>> This seems like an
>> easy enough thing, but I am consistently getting hit with
>> ClassNotFoundException which
>> makes no sense to me. Any help would be immensely appreciated. Note
>> that I've had
>> valueaggregators running on 0.13.*. I'm a little concerned there is a bug
>> in 0.14.1. On the
>> other hand, maybe I'm just blind to something obvious and just need a
>> boot to the head...
>>
>> The environment is hadoop-0.14.1. I am running Java 1.6:
>> $ java -version
>> java version "1.6.0_02"
>> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
>> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
>>
>> The class name is called testAggregator, and exists in my jar file:
>> $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
>> 1826 Fri Oct 12 16:30:14 EDT 2007
>> com/company/hadoop/metrics/testAggregator.class
>>
>> My XML file is testAggregator.xml and looks like:
>>
>>
>>
>>
>
>> aggregator.descriptor.num
>> 1
>>
>
>
>> aggregator.descriptor.0
>> UserDefined,com.company.hadoop.metrics.testAggregator
>>
>
>>
>> My code couldn't be simpler:
>>
>> package com.company.hadoop.metrics;
>> // snip a bunch of imports
>> public class testAggregator extends ValueAggregatorBaseDescriptor {
>> public ArrayList generateKeyValPairs(Object key, Object val) {
>> String fields[] = val.toString().split("\t");
>> ArrayList retv = new ArrayList();
>> retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
>> retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
>> fields[ 2 ])));
>> return retv;
>> }
>> public void configure(JobConf job) {}
>> }
>>
>> I invoke things this way:
>>
>> bin/hadoop jar hadoop-0.14.1-dev-company.jar
>> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
>> /output/set1 8 textinputformat testAggregator.xml
>>
>> All my jobs fail with a ClassNotFoundException like the following
>> traceback:
>>
>> 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=MAP, sessionId=
>> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
>> numReduceTasks: 8
>> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
>> running child
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> com.company.hadoop.metrics.testAggregator
>> at
>>
>> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
>> r.createInstance
>> (UserDefinedValueAggregatorDescriptor.java:56)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
>> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
>> r.(UserDefinedValueAggregatorDescriptor.java:75)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
>> egatorDescriptor(ValueAggregatorJobBase.java:49)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
>> rDescriptors(ValueAggregatorJobBase.java:60)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
>> Spec(ValueAggregatorJobBase.java:69)
>> at
>> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
>> lueAggregatorJobBase.java:37)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> )
>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
>> )
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
>> at
>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
>> Caused by: java.lang.ClassNotFoundException:
>> com.company.hadoop.metrics.testAggregator
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at
>>
>> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
>> r.createInstance
>> (UserDefinedValueAggregatorDescriptor.java:51)
>> ... 13 more
>>
>> As near as I can tell I am doing everything correctly. Other code
>> existing in this jar file is
>> running just fine - in fact it was used to produce the input to this
>> test. I've checked my
>> systems to make I've got the jar file (and XML file) onto each of the 4
>> compute nodes in the
>> system.
>>
>> Can anybody spot my problem or tell me what to do to get this going?
>>
>> Thanks,
>> C G
>>
>>
>> ---------------------------------
>> Tonight's top picks. What will you watch tonight? Preview the hottest
>> shows on Yahoo! TV.
>
>
>
>
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who knows.
> Yahoo! Answers - Check it out.
RE: Question about valueaggregators in 0.14.1...
Posted by C G <pa...@yahoo.com>.
Hi Runping and All:
I took your suggestion, renamed my original code to testAggregatorPlugin and wrote a class testAggregator which contained the main as you show here. It also fails with the same traceback...so it seems something is profoundly wrong here.
Is anybody else using the aggregation classes in 0.14.1 with success?
This is ironic and frustrating...I have several other programs which use map/reduce that are quite complex and those are running just fine. This little plugin is all that stands in the way of delivering the system :-(.
Any other comments/help, etc. most welcome...
Thanks,
C G
Runping Qi <ru...@yahoo-inc.com> wrote:
Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
In the meantime, you can create a class like below and add it to your jar:
package your_package;
public class MyValueAggregatorJob{
public static void main(String args[]) throws IOException {
JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
JobClient.runJob(job);
}
}
And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml
Hope this solve your problem.
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 2:43 PM
> To: hadoop-user@lucene.apache.org
> Subject: Question about valueaggregators in 0.14.1...
>
> Hi All:
>
> I am tearing out my hair trying to get a simple valueaggregator to run.
> This seems like an
> easy enough thing, but I am consistently getting hit with
> ClassNotFoundException which
> makes no sense to me. Any help would be immensely appreciated. Note
> that I've had
> valueaggregators running on 0.13.*. I'm a little concerned there is a bug
> in 0.14.1. On the
> other hand, maybe I'm just blind to something obvious and just need a
> boot to the head...
>
> The environment is hadoop-0.14.1. I am running Java 1.6:
> $ java -version
> java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
>
> The class name is called testAggregator, and exists in my jar file:
> $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> 1826 Fri Oct 12 16:30:14 EDT 2007
> com/company/hadoop/metrics/testAggregator.class
>
> My XML file is testAggregator.xml and looks like:
>
>
>
>
> aggregator.descriptor.num
> 1
>
>
> aggregator.descriptor.0
> UserDefined,com.company.hadoop.metrics.testAggregator
>
>
>
> My code couldn't be simpler:
>
> package com.company.hadoop.metrics;
> // snip a bunch of imports
> public class testAggregator extends ValueAggregatorBaseDescriptor {
> public ArrayList generateKeyValPairs(Object key, Object val) {
> String fields[] = val.toString().split("\t");
> ArrayList retv = new ArrayList();
> retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> fields[ 2 ])));
> return retv;
> }
> public void configure(JobConf job) {}
> }
>
> I invoke things this way:
>
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> /output/set1 8 textinputformat testAggregator.xml
>
> All my jobs fail with a ClassNotFoundException like the following
> traceback:
>
> 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 8
> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:56)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.(UserDefinedValueAggregatorDescriptor.java:75)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> egatorDescriptor(ValueAggregatorJobBase.java:49)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> rDescriptors(ValueAggregatorJobBase.java:60)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> Spec(ValueAggregatorJobBase.java:69)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> lueAggregatorJobBase.java:37)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> Caused by: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:51)
> ... 13 more
>
> As near as I can tell I am doing everything correctly. Other code
> existing in this jar file is
> running just fine - in fact it was used to produce the input to this
> test. I've checked my
> systems to make I've got the jar file (and XML file) onto each of the 4
> compute nodes in the
> system.
>
> Can anybody spot my problem or tell me what to do to get this going?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Tonight's top picks. What will you watch tonight? Preview the hottest
> shows on Yahoo! TV.
---------------------------------
Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out.
RE: Question about valueaggregators in 0.14.1...
Posted by Runping Qi <ru...@yahoo-inc.com>.
Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
In the meantime, you can create a class like below and add it to your jar:
package your_package;
public class MyValueAggregatorJob{
public static void main(String args[]) throws IOException {
JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
JobClient.runJob(job);
}
}
And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml
Hope this solve your problem.
Runping
> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 2:43 PM
> To: hadoop-user@lucene.apache.org
> Subject: Question about valueaggregators in 0.14.1...
>
> Hi All:
>
> I am tearing out my hair trying to get a simple valueaggregator to run.
> This seems like an
> easy enough thing, but I am consistently getting hit with
> ClassNotFoundException which
> makes no sense to me. Any help would be immensely appreciated. Note
> that I've had
> valueaggregators running on 0.13.*. I'm a little concerned there is a bug
> in 0.14.1. On the
> other hand, maybe I'm just blind to something obvious and just need a
> boot to the head...
>
> The environment is hadoop-0.14.1. I am running Java 1.6:
> $ java -version
> java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
>
> The class name is called testAggregator, and exists in my jar file:
> $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> 1826 Fri Oct 12 16:30:14 EDT 2007
> com/company/hadoop/metrics/testAggregator.class
>
> My XML file is testAggregator.xml and looks like:
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
> <property>
> <name>aggregator.descriptor.num</name>
> <value>1</value>
> </property>
> <property>
> <name>aggregator.descriptor.0</name>
> <value>UserDefined,com.company.hadoop.metrics.testAggregator</value>
> </property>
> </configuration>
>
> My code couldn't be simpler:
>
> package com.company.hadoop.metrics;
> // snip a bunch of imports
> public class testAggregator extends ValueAggregatorBaseDescriptor {
> public ArrayList<Entry> generateKeyValPairs(Object key, Object val) {
> String fields[] = val.toString().split("\t");
> ArrayList<Entry> retv = new ArrayList<Entry>();
> retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> fields[ 2 ])));
> return retv;
> }
> public void configure(JobConf job) {}
> }
>
> I invoke things this way:
>
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> /output/set1 8 textinputformat testAggregator.xml
>
> All my jobs fail with a ClassNotFoundException like the following
> traceback:
>
> 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 8
> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:56)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.<init>(UserDefinedValueAggregatorDescriptor.java:75)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> egatorDescriptor(ValueAggregatorJobBase.java:49)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> rDescriptors(ValueAggregatorJobBase.java:60)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> Spec(ValueAggregatorJobBase.java:69)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> lueAggregatorJobBase.java:37)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> Caused by: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:51)
> ... 13 more
>
> As near as I can tell I am doing everything correctly. Other code
> existing in this jar file is
> running just fine - in fact it was used to produce the input to this
> test. I've checked my
> systems to make I've got the jar file (and XML file) onto each of the 4
> compute nodes in the
> system.
>
> Can anybody spot my problem or tell me what to do to get this going?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Tonight's top picks. What will you watch tonight? Preview the hottest
> shows on Yahoo! TV.