You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by imen Megdiche <im...@gmail.com> on 2012/12/12 10:23:39 UTC
Number of mapreduce job and the time spent
Hi,
I want to know from the output of the execution of the example of
mapreduce wordcount on hadoop : the number of mapreduce job and the time
spent for the execution.
There is an exceprt from the output.
12/12/12 10:20:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
done.
12/12/12 10:20:10 INFO mapred.JobClient: map 100% reduce 100%
12/12/12 10:20:10 INFO mapred.JobClient: Job complete: job_local_0001
12/12/12 10:20:10 INFO mapred.JobClient: Counters: 22
12/12/12 10:20:10 INFO mapred.JobClient: File Input Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Read=145966941
12/12/12 10:20:10 INFO mapred.JobClient: File Output Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Written=50704638
12/12/12 10:20:10 INFO mapred.JobClient: org.myorg.WordCount$Map$Counters
12/12/12 10:20:10 INFO mapred.JobClient: INPUT_WORDS=4980060
12/12/12 10:20:10 INFO mapred.JobClient: FileSystemCounters
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_READ=1777104865
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1783494521
12/12/12 10:20:10 INFO mapred.JobClient: Map-Reduce Framework
12/12/12 10:20:10 INFO mapred.JobClient: Map output materialized
bytes=170854986
12/12/12 10:20:10 INFO mapred.JobClient: Map input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce shuffle bytes=0
12/12/12 10:20:10 INFO mapred.JobClient: Spilled Records=14940180
12/12/12 10:20:10 INFO mapred.JobClient: Map output bytes=160894830
12/12/12 10:20:10 INFO mapred.JobClient: Total committed heap usage
(bytes)=1185910784
12/12/12 10:20:10 INFO mapred.JobClient: CPU time spent (ms)=0
12/12/12 10:20:10 INFO mapred.JobClient: Map input bytes=145954650
12/12/12 10:20:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=614
12/12/12 10:20:10 INFO mapred.JobClient: Combine input records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input groups=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Combine output records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Physical memory (bytes)
snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Reduce output records=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Map output records=4980060
Thank you for your responses.
Re: Re: Number of mapreduce job and the time spent
Posted by imen Megdiche <im...@gmail.com>.
the command hadoop job -status work fine but the problem that it cannot
find the job
Could not find job job_local_0001
i don t understand why does it not find it.
2012/12/12 long <lo...@163.com>
> Sorry for my mistake.
> if $HADOOP_HOME is set, run as follow, or not just find the path for your
> 'hadoop' command for instead:
> $HADOOP_HOME/bin/hadoop job -status job_xxx
>
>
>
>
> --
> Best Regards,
> longmans
>
> At 2012-12-12 17:56:45,"imen Megdiche" <im...@gmail.com> wrote:
>
> I think that my job id is in this line :
>
> 12/12/12 10:43:00 INFO mapred.JobClient: Running job: job_local_0001
>
>
> but i have this response when i execute :
>
> hadoop job -status job_local_0001
> Warning: $HADOOP_HOME is deprecated.
>
> Could not find job job_local_0001
>
>
>
>
>
> 2012/12/12 long <lo...@163.com>
>
>> get you jobid and use this command:
>> $HADOOP_HOME/hadoop job -status job_xxx
>>
>>
>>
>>
>> --
>> Best Regards,
>> longmans
>>
>> At 2012-12-12 17:23:39,"imen Megdiche" <im...@gmail.com> wrote:
>>
>> Hi,
>>
>> I want to know from the output of the execution of the example of
>> mapreduce wordcount on hadoop : the number of mapreduce job and the time
>> spent for the execution.
>>
>> There is an exceprt from the output.
>>
>> 12/12/12 10:20:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
>> done.
>> 12/12/12 10:20:10 INFO mapred.JobClient: map 100% reduce 100%
>> 12/12/12 10:20:10 INFO mapred.JobClient: Job complete: job_local_0001
>> 12/12/12 10:20:10 INFO mapred.JobClient: Counters: 22
>> 12/12/12 10:20:10 INFO mapred.JobClient: File Input Format Counters
>> 12/12/12 10:20:10 INFO mapred.JobClient: Bytes Read=145966941
>> 12/12/12 10:20:10 INFO mapred.JobClient: File Output Format Counters
>> 12/12/12 10:20:10 INFO mapred.JobClient: Bytes Written=50704638
>> 12/12/12 10:20:10 INFO mapred.JobClient:
>> org.myorg.WordCount$Map$Counters
>> 12/12/12 10:20:10 INFO mapred.JobClient: INPUT_WORDS=4980060
>> 12/12/12 10:20:10 INFO mapred.JobClient: FileSystemCounters
>> 12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_READ=1777104865
>> 12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1783494521
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map-Reduce Framework
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map output materialized
>> bytes=170854986
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map input records=4980060
>> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce shuffle bytes=0
>> 12/12/12 10:20:10 INFO mapred.JobClient: Spilled Records=14940180
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map output bytes=160894830
>> 12/12/12 10:20:10 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1185910784
>> 12/12/12 10:20:10 INFO mapred.JobClient: CPU time spent (ms)=0
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map input bytes=145954650
>> 12/12/12 10:20:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=614
>> 12/12/12 10:20:10 INFO mapred.JobClient: Combine input records=8426541
>> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce input records=4980060
>> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce input groups=1660020
>> 12/12/12 10:20:10 INFO mapred.JobClient: Combine output
>> records=8426541
>> 12/12/12 10:20:10 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=0
>> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce output records=1660020
>> 12/12/12 10:20:10 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=0
>> 12/12/12 10:20:10 INFO mapred.JobClient: Map output records=4980060
>>
>>
>> Thank you for your responses.
>>
>>
>>
>>
>>
>
>
>
Re:Re: Number of mapreduce job and the time spent
Posted by long <lo...@163.com>.
Sorry for my mistake.
if $HADOOP_HOME is set, run as follow, or not just find the path for your 'hadoop' command for instead:
$HADOOP_HOME/bin/hadoop job -status job_xxx
--
Best Regards,
longmans
At 2012-12-12 17:56:45,"imen Megdiche" <im...@gmail.com> wrote:
I think that my job id is in this line :
12/12/12 10:43:00 INFO mapred.JobClient: Running job: job_local_0001
but i have this response when i execute :
hadoop job -status job_local_0001
Warning: $HADOOP_HOME is deprecated.
Could not find job job_local_0001
2012/12/12 long <lo...@163.com>
get you jobid and use this command:
$HADOOP_HOME/hadoop job -status job_xxx
--
Best Regards,
longmans
At 2012-12-12 17:23:39,"imen Megdiche" <im...@gmail.com> wrote:
Hi,
I want to know from the output of the execution of the example of mapreduce wordcount on hadoop : the number of mapreduce job and the time spent for the execution.
There is an exceprt from the output.
12/12/12 10:20:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/12/12 10:20:10 INFO mapred.JobClient: map 100% reduce 100%
12/12/12 10:20:10 INFO mapred.JobClient: Job complete: job_local_0001
12/12/12 10:20:10 INFO mapred.JobClient: Counters: 22
12/12/12 10:20:10 INFO mapred.JobClient: File Input Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Read=145966941
12/12/12 10:20:10 INFO mapred.JobClient: File Output Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Written=50704638
12/12/12 10:20:10 INFO mapred.JobClient: org.myorg.WordCount$Map$Counters
12/12/12 10:20:10 INFO mapred.JobClient: INPUT_WORDS=4980060
12/12/12 10:20:10 INFO mapred.JobClient: FileSystemCounters
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_READ=1777104865
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1783494521
12/12/12 10:20:10 INFO mapred.JobClient: Map-Reduce Framework
12/12/12 10:20:10 INFO mapred.JobClient: Map output materialized bytes=170854986
12/12/12 10:20:10 INFO mapred.JobClient: Map input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce shuffle bytes=0
12/12/12 10:20:10 INFO mapred.JobClient: Spilled Records=14940180
12/12/12 10:20:10 INFO mapred.JobClient: Map output bytes=160894830
12/12/12 10:20:10 INFO mapred.JobClient: Total committed heap usage (bytes)=1185910784
12/12/12 10:20:10 INFO mapred.JobClient: CPU time spent (ms)=0
12/12/12 10:20:10 INFO mapred.JobClient: Map input bytes=145954650
12/12/12 10:20:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=614
12/12/12 10:20:10 INFO mapred.JobClient: Combine input records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input groups=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Combine output records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Reduce output records=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Map output records=4980060
Thank you for your responses.
Re: Number of mapreduce job and the time spent
Posted by imen Megdiche <im...@gmail.com>.
I think that my job id is in this line :
12/12/12 10:43:00 INFO mapred.JobClient: Running job: job_local_0001
but i have this response when i execute :
hadoop job -status job_local_0001
Warning: $HADOOP_HOME is deprecated.
Could not find job job_local_0001
2012/12/12 long <lo...@163.com>
> get you jobid and use this command:
> $HADOOP_HOME/hadoop job -status job_xxx
>
>
>
>
> --
> Best Regards,
> longmans
>
> At 2012-12-12 17:23:39,"imen Megdiche" <im...@gmail.com> wrote:
>
> Hi,
>
> I want to know from the output of the execution of the example of
> mapreduce wordcount on hadoop : the number of mapreduce job and the time
> spent for the execution.
>
> There is an exceprt from the output.
>
> 12/12/12 10:20:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
> done.
> 12/12/12 10:20:10 INFO mapred.JobClient: map 100% reduce 100%
> 12/12/12 10:20:10 INFO mapred.JobClient: Job complete: job_local_0001
> 12/12/12 10:20:10 INFO mapred.JobClient: Counters: 22
> 12/12/12 10:20:10 INFO mapred.JobClient: File Input Format Counters
> 12/12/12 10:20:10 INFO mapred.JobClient: Bytes Read=145966941
> 12/12/12 10:20:10 INFO mapred.JobClient: File Output Format Counters
> 12/12/12 10:20:10 INFO mapred.JobClient: Bytes Written=50704638
> 12/12/12 10:20:10 INFO mapred.JobClient: org.myorg.WordCount$Map$Counters
> 12/12/12 10:20:10 INFO mapred.JobClient: INPUT_WORDS=4980060
> 12/12/12 10:20:10 INFO mapred.JobClient: FileSystemCounters
> 12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_READ=1777104865
> 12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1783494521
> 12/12/12 10:20:10 INFO mapred.JobClient: Map-Reduce Framework
> 12/12/12 10:20:10 INFO mapred.JobClient: Map output materialized
> bytes=170854986
> 12/12/12 10:20:10 INFO mapred.JobClient: Map input records=4980060
> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce shuffle bytes=0
> 12/12/12 10:20:10 INFO mapred.JobClient: Spilled Records=14940180
> 12/12/12 10:20:10 INFO mapred.JobClient: Map output bytes=160894830
> 12/12/12 10:20:10 INFO mapred.JobClient: Total committed heap usage
> (bytes)=1185910784
> 12/12/12 10:20:10 INFO mapred.JobClient: CPU time spent (ms)=0
> 12/12/12 10:20:10 INFO mapred.JobClient: Map input bytes=145954650
> 12/12/12 10:20:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=614
> 12/12/12 10:20:10 INFO mapred.JobClient: Combine input records=8426541
> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce input records=4980060
> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce input groups=1660020
> 12/12/12 10:20:10 INFO mapred.JobClient: Combine output records=8426541
> 12/12/12 10:20:10 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=0
> 12/12/12 10:20:10 INFO mapred.JobClient: Reduce output records=1660020
> 12/12/12 10:20:10 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=0
> 12/12/12 10:20:10 INFO mapred.JobClient: Map output records=4980060
>
>
> Thank you for your responses.
>
>
>
>
>
Re:Number of mapreduce job and the time spent
Posted by long <lo...@163.com>.
get you jobid and use this command:
$HADOOP_HOME/hadoop job -status job_xxx
--
Best Regards,
longmans
At 2012-12-12 17:23:39,"imen Megdiche" <im...@gmail.com> wrote:
Hi,
I want to know from the output of the execution of the example of mapreduce wordcount on hadoop : the number of mapreduce job and the time spent for the execution.
There is an exceprt from the output.
12/12/12 10:20:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/12/12 10:20:10 INFO mapred.JobClient: map 100% reduce 100%
12/12/12 10:20:10 INFO mapred.JobClient: Job complete: job_local_0001
12/12/12 10:20:10 INFO mapred.JobClient: Counters: 22
12/12/12 10:20:10 INFO mapred.JobClient: File Input Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Read=145966941
12/12/12 10:20:10 INFO mapred.JobClient: File Output Format Counters
12/12/12 10:20:10 INFO mapred.JobClient: Bytes Written=50704638
12/12/12 10:20:10 INFO mapred.JobClient: org.myorg.WordCount$Map$Counters
12/12/12 10:20:10 INFO mapred.JobClient: INPUT_WORDS=4980060
12/12/12 10:20:10 INFO mapred.JobClient: FileSystemCounters
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_READ=1777104865
12/12/12 10:20:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1783494521
12/12/12 10:20:10 INFO mapred.JobClient: Map-Reduce Framework
12/12/12 10:20:10 INFO mapred.JobClient: Map output materialized bytes=170854986
12/12/12 10:20:10 INFO mapred.JobClient: Map input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce shuffle bytes=0
12/12/12 10:20:10 INFO mapred.JobClient: Spilled Records=14940180
12/12/12 10:20:10 INFO mapred.JobClient: Map output bytes=160894830
12/12/12 10:20:10 INFO mapred.JobClient: Total committed heap usage (bytes)=1185910784
12/12/12 10:20:10 INFO mapred.JobClient: CPU time spent (ms)=0
12/12/12 10:20:10 INFO mapred.JobClient: Map input bytes=145954650
12/12/12 10:20:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=614
12/12/12 10:20:10 INFO mapred.JobClient: Combine input records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input records=4980060
12/12/12 10:20:10 INFO mapred.JobClient: Reduce input groups=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Combine output records=8426541
12/12/12 10:20:10 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Reduce output records=1660020
12/12/12 10:20:10 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
12/12/12 10:20:10 INFO mapred.JobClient: Map output records=4980060
Thank you for your responses.