You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ANKITBHATNAGAR <ab...@vantage.com> on 2010/11/01 15:56:53 UTC
Re: Hadoop Reduce is never called
This is looks suspicious -
10/10/30 11:54:08 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Ankit
--
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30105474.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Hadoop Reduce is never called
Posted by Cameron_ <rc...@sfu.ca>.
Thanks for your help - I found the problem. The signature for my reduce
function had Iterator instead of Iterable. Without the right signature, it
was calling the reduce method from the superclass and not mine.
ANKITBHATNAGAR wrote:
>
> Also just noticed that your extension is not generic ?
>
> extends Reducer{
>
>
> should be extends Reducer<...>{
>
>
>
>
--
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30117511.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Hadoop Reduce is never called
Posted by ANKITBHATNAGAR <ab...@vantage.com>.
Also just noticed that your extension is not generic ?
extends Reducer{
should be extends Reducer<...>{
--
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30116358.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Hadoop Reduce is never called
Posted by ANKITBHATNAGAR <ab...@vantage.com>.
10/10/30 11:54:10 INFO mapred.JobClient: Reduce output records=32242
10/10/30 11:54:10 INFO mapred.JobClient: Spilled Records=64484
10/10/30 11:54:10 INFO mapred.JobClient: Map output bytes=1840378
10/10/30 11:54:10 INFO mapred.JobClient: Combine input records=0
10/10/30 11:54:10 INFO mapred.JobClient: Map output records=32242
10/10/30 11:54:10 INFO mapred.JobClient: Reduce input records=32242
These bold lines suggest that infact reduce was called.
Could you debug and check the reducer?
--
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30116293.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Hadoop Reduce is never called
Posted by Cameron_ <rc...@sfu.ca>.
The deprecated warning is not the issue - This message appears only because I
am running it in an eclipse environment - when run from the command line it
is not there.
ANKITBHATNAGAR wrote:
>
> This is looks suspicious -
>
> 10/10/30 11:54:08 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
> use core-site.xml, mapred-site.xml and hdfs-site.xml to override
> properties of core-default.xml, mapred-default.xml and hdfs-default.xml
> respectively
>
>
> Ankit
>
--
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30115480.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.