You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by CTS-RAAJ <ra...@cognizant.com> on 2008/07/04 14:51:23 UTC
Failed to get reducer output while running in cluster
Hi All,
After all successful work aroud in getting the hadoop cluster run, I am
kicked back by some problem while executig my program in cluster,
Program takes input a log file, I have written a mapper which goes through
the line to find certain pattern and pass the mapper output to reducer to
sort and give me the counters. This I pass to a small job, which creates a
chart using apache poi and itext api.
When I execute locally it runs fine with follwoing output
LOCAL
08/07/04 17:24:23 INFO mapred.LocalJobRunner:
file:/home/rajesh/POC/hadoop-0.17.0/sdeinput1/SDE_LOG.log:0+27131
08/07/04 17:24:23 INFO mapred.TaskRunner: Task 'job_local_1_map_0000' done.
08/07/04 17:24:23 INFO mapred.TaskRunner: Saved output of task
'job_local_1_map_0000' to file:/home/rajesh/POC/hadoop-0.17.0/output1
08/07/04 17:24:23 INFO mapred.LocalJobRunner: reduce > reduce
08/07/04 17:24:23 INFO mapred.TaskRunner: Task 'reduce_75lmh0' done.
08/07/04 17:24:23 INFO mapred.TaskRunner: Saved output of task
'reduce_75lmh0' to file:/home/rajesh/POC/hadoop-0.17.0/output108/07/04
17:24:23 INFO mapred.JobClient: Job complete: job_local_1
08/07/04 17:24:23 INFO mapred.JobClient: Counters: 11
08/07/04 17:24:23 INFO mapred.JobClient: Map-Reduce Framework
08/07/04 17:24:23 INFO mapred.JobClient: Map input records=316
08/07/04 17:24:23 INFO mapred.JobClient: Map output records=102
08/07/04 17:24:23 INFO mapred.JobClient: Map input bytes=27131
08/07/04 17:24:23 INFO mapred.JobClient: Map output bytes=816
08/07/04 17:24:23 INFO mapred.JobClient: Combine input records=102
08/07/04 17:24:23 INFO mapred.JobClient: Combine output records=3
08/07/04 17:24:23 INFO mapred.JobClient: Reduce input groups=3
08/07/04 17:24:23 INFO mapred.JobClient: Reduce input records=3
08/07/04 17:24:23 INFO mapred.JobClient: Reduce output records=3
08/07/04 17:24:23 INFO mapred.JobClient: File Systems
08/07/04 17:24:23 INFO mapred.JobClient: Local bytes read=110873
08/07/04 17:24:23 INFO mapred.JobClient: Local bytes written=83356
:::::Just after the job conf call:::::
Here I create the pdf file
And my cluster output, please check the bold part
CLUSTER
[rajesh@master hadoop-0.17.0]$ ./bin/hadoop jar hadoop-sdepoc-cluster1.jar
MyWordCount clsinput1 output2 june
08/07/04 17:48:25 INFO mapred.FileInputFormat: Total input paths to process
: 1
08/07/04 17:48:25 INFO mapred.JobClient: Running job: job_200807041747_0001
08/07/04 17:48:26 INFO mapred.JobClient: map 0% reduce 0%
08/07/04 17:48:31 INFO mapred.JobClient: map 50% reduce 0%
08/07/04 17:48:33 INFO mapred.JobClient: map 100% reduce 0%
08/07/04 17:48:36 INFO mapred.JobClient: map 100% reduce 100%
08/07/04 17:48:37 INFO mapred.JobClient: Job complete: job_200807041747_0001
08/07/04 17:48:37 INFO mapred.JobClient: Counters: 15
08/07/04 17:48:37 INFO mapred.JobClient: Job Counters
08/07/04 17:48:37 INFO mapred.JobClient: Launched map tasks=2
08/07/04 17:48:37 INFO mapred.JobClient: Launched reduce tasks=1
08/07/04 17:48:37 INFO mapred.JobClient: Data-local map tasks=2
08/07/04 17:48:37 INFO mapred.JobClient: Map-Reduce Framework
08/07/04 17:48:37 INFO mapred.JobClient: Map input records=316
08/07/04 17:48:37 INFO mapred.JobClient: Map output records=0
08/07/04 17:48:37 INFO mapred.JobClient: Map input bytes=27131
08/07/04 17:48:37 INFO mapred.JobClient: Map output bytes=0
08/07/04 17:48:37 INFO mapred.JobClient: Combine input records=0
08/07/04 17:48:37 INFO mapred.JobClient: Combine output records=0
08/07/04 17:48:37 INFO mapred.JobClient: Reduce input groups=0
08/07/04 17:48:37 INFO mapred.JobClient: Reduce input records=0
08/07/04 17:48:37 INFO mapred.JobClient: Reduce output records=0
08/07/04 17:48:37 INFO mapred.JobClient: File Systems
08/07/04 17:48:37 INFO mapred.JobClient: Local bytes read=117
08/07/04 17:48:37 INFO mapred.JobClient: Local bytes written=367
08/07/04 17:48:37 INFO mapred.JobClient: HDFS bytes read=29951
:::::Just after the job conf call:::::
[rajesh@master hadoop-0.17.0]$
Please suggest me some approach or inform if code needs to be uploaded.
--
View this message in context: http://www.nabble.com/Failed-to-get-reducer-output-while-running-in-cluster-tp18278864p18278864.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: NEED AN URGENT HELP PL, Failed to get reducer output while
running in cluster
Posted by CTS-RAAJ <ra...@cognizant.com>.
CTS-RAAJ wrote:
>
> Hi All,
>
> After all successful work aroud in getting the hadoop cluster run, I am
> kicked back by some problem while executig my program in cluster,
> Program takes input a log file, I have written a mapper which goes through
> the line to find certain pattern and pass the mapper output to reducer to
> sort and give me the counters. This I pass to a small job, which creates a
> chart using apache poi and itext api.
>
> When I execute locally it runs fine with follwoing output
>
> LOCAL
> 08/07/04 17:24:23 INFO mapred.LocalJobRunner:
> file:/home/rajesh/POC/hadoop-0.17.0/sdeinput1/SDE_LOG.log:0+27131
> 08/07/04 17:24:23 INFO mapred.TaskRunner: Task 'job_local_1_map_0000'
> done.
> 08/07/04 17:24:23 INFO mapred.TaskRunner: Saved output of task
> 'job_local_1_map_0000' to file:/home/rajesh/POC/hadoop-0.17.0/output1
> 08/07/04 17:24:23 INFO mapred.LocalJobRunner: reduce > reduce
> 08/07/04 17:24:23 INFO mapred.TaskRunner: Task 'reduce_75lmh0' done.
> 08/07/04 17:24:23 INFO mapred.TaskRunner: Saved output of task
> 'reduce_75lmh0' to file:/home/rajesh/POC/hadoop-0.17.0/output108/07/04
> 17:24:23 INFO mapred.JobClient: Job complete: job_local_1
> 08/07/04 17:24:23 INFO mapred.JobClient: Counters: 11
> 08/07/04 17:24:23 INFO mapred.JobClient: Map-Reduce Framework
> 08/07/04 17:24:23 INFO mapred.JobClient: Map input records=316
> 08/07/04 17:24:23 INFO mapred.JobClient: Map output records=102
> 08/07/04 17:24:23 INFO mapred.JobClient: Map input bytes=27131
> 08/07/04 17:24:23 INFO mapred.JobClient: Map output bytes=816
> 08/07/04 17:24:23 INFO mapred.JobClient: Combine input records=102
> 08/07/04 17:24:23 INFO mapred.JobClient: Combine output records=3
> 08/07/04 17:24:23 INFO mapred.JobClient: Reduce input groups=3
> 08/07/04 17:24:23 INFO mapred.JobClient: Reduce input records=3
> 08/07/04 17:24:23 INFO mapred.JobClient: Reduce output records=3
> 08/07/04 17:24:23 INFO mapred.JobClient: File Systems
> 08/07/04 17:24:23 INFO mapred.JobClient: Local bytes read=110873
> 08/07/04 17:24:23 INFO mapred.JobClient: Local bytes written=83356
> :::::Just after the job conf call:::::
> Here I create the pdf file
>
>
> And my cluster output, please check the bold part
>
> CLUSTER
> [rajesh@master hadoop-0.17.0]$ ./bin/hadoop jar hadoop-sdepoc-cluster1.jar
> MyWordCount clsinput1 output2 june
> 08/07/04 17:48:25 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 08/07/04 17:48:25 INFO mapred.JobClient: Running job:
> job_200807041747_0001
> 08/07/04 17:48:26 INFO mapred.JobClient: map 0% reduce 0%
> 08/07/04 17:48:31 INFO mapred.JobClient: map 50% reduce 0%
> 08/07/04 17:48:33 INFO mapred.JobClient: map 100% reduce 0%
> 08/07/04 17:48:36 INFO mapred.JobClient: map 100% reduce 100%
> 08/07/04 17:48:37 INFO mapred.JobClient: Job complete:
> job_200807041747_0001
> 08/07/04 17:48:37 INFO mapred.JobClient: Counters: 15
> 08/07/04 17:48:37 INFO mapred.JobClient: Job Counters
> 08/07/04 17:48:37 INFO mapred.JobClient: Launched map tasks=2
> 08/07/04 17:48:37 INFO mapred.JobClient: Launched reduce tasks=1
> 08/07/04 17:48:37 INFO mapred.JobClient: Data-local map tasks=2
> 08/07/04 17:48:37 INFO mapred.JobClient: Map-Reduce Framework
> 08/07/04 17:48:37 INFO mapred.JobClient: Map input records=316
> 08/07/04 17:48:37 INFO mapred.JobClient: Map output records=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Map input bytes=27131
> 08/07/04 17:48:37 INFO mapred.JobClient: Map output bytes=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Combine input records=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Combine output records=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Reduce input groups=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Reduce input records=0
> 08/07/04 17:48:37 INFO mapred.JobClient: Reduce output records=0
> 08/07/04 17:48:37 INFO mapred.JobClient: File Systems
> 08/07/04 17:48:37 INFO mapred.JobClient: Local bytes read=117
> 08/07/04 17:48:37 INFO mapred.JobClient: Local bytes written=367
> 08/07/04 17:48:37 INFO mapred.JobClient: HDFS bytes read=29951
> :::::Just after the job conf call:::::
> [rajesh@master hadoop-0.17.0]$
>
> Please suggest me some approach or inform if code needs to be uploaded.
>
>
:working::working::working::working:
--
View this message in context: http://www.nabble.com/Failed-to-get-reducer-output-while-running-in-cluster-tp18278864p18292904.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.