Re: Inconsistency while running in eclipse and cygwin

[Sridhar Raman <sr...@gmail.com>]

I managed to fix both the problems.

1)  The one in Eclipse was happening because of the df task which wasn't
possible in Windows.  Once I added cygwin/bin to the Path, this started
working.

2)  This one occurred because the output value class of my Combiner and
Reducer was different.

On Wed, Apr 30, 2008 at 6:49 PM, Sridhar Raman <sr...@gmail.com>
wrote:

> I am trying to run my MR task in a local machine.  I am doing this because
> this is my first foray into MapReduce, and want to debug completely before I
> move it to a multi-node cluser.  The initial hitch that I faced was in
> running it from Eclipse.  Thanks to this response<http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg01368.html>,
> I managed to get that working.
>
> But now I am facing a strange problem.
>
> My Mapper takes <IntWritable, Vector> as input, and returns <IntWritable,
> Vector>.
> I have a Combiner that takes the <IntWritable, Iterator<Vector>> and
> computes the intermediate output of <IntWritable, InterVector>.  This
> IntVector basically stores the sum of the Vectors present in the iterator,
> along with other information.
>
> Now to the problem.  If I run this in Eclipse, after the Mapper is done, I
> straightaway get this exception:
> Iteration 0
> java.io.IOException: Job failed!
>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.runIteration(KMeansDriver.java:144)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.runJob(KMeansDriver.java:91)
>     at
> com.company.analytics.clustering.mr.core.KMeansDriver.main(KMeansDriver.java:37)
>
> Whereas if I run this from the cygwin command prompt, the execution
> manages to go to the Combiner, but after that it gives this exception:
> java.io.IOException: wrong value class:
> com.company.analytics.clustering.feature.
> InterVector is not class com.company.analytics.clustering.feature.Vector
>         at
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java:414)
>         at
> com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:57)
>         at
> com.company.analytics.clustering.mr.core.KMeansCombiner.reduce(KMeansCombiner.java:1)
>
> Any idea what could be the problem.  Both the cygwin and Eclipse take the
> same input folder.
> I would be glad if I get any assistance.
>
> Thanks,
> Sridhar
>