You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Han Dong <ha...@gmail.com> on 2010/09/25 20:11:01 UTC
Log file questions
Hi,
I have a question regarding the log files generated when running some simple
hadoop examples in the jar file I was given. The log files contained the
time it took to run the map and reduce tasks, however, does this time also
include I/O? If I were to run the wordcount example, it would need to read
in the file at some point, does the time of the map and reduce tasks
generated in the log files include that timing as well?
Below is an example of the log file generated for a map task.
Task
Complete
Status
Start Time
Finish Time
Errors
Counters
task_201009071545_0028_m_000000<http://172.20.101.33:50030/taskdetails.jsp?jobid=job_201009071545_0028&tipid=task_201009071545_0028_m_000000>
100.00%
Finished processing 30300 records from the input file:
hdfs://blade33:9000/user/han6/InputFile/5MB.txt
21-Sep-2010 12:45:17
21-Sep-2010 12:45:32 (15sec)
11<http://172.20.101.33:50030/taskstats.jsp?jobid=job_201009071545_0028&tipid=task_201009071545_0028_m_000000>
Thanks,
Han Dong
handong32@gmail.com
Re: Log file questions
Posted by Mario M <ma...@gmail.com>.
Hi,
it doesn't include the time needed to divide the input into splits for each
map task, that I can tell you for sure (e.g. my program takes 1 minute
processing and 30 minutes dividing the input, but the log only shows one
minute).
Mario M
2010/9/25 Han Dong <ha...@gmail.com>
> Hi,
>
> I have a question regarding the log files generated when running some
> simple hadoop examples in the jar file I was given. The log files contained
> the time it took to run the map and reduce tasks, however, does this time
> also include I/O? If I were to run the wordcount example, it would need to
> read in the file at some point, does the time of the map and reduce tasks
> generated in the log files include that timing as well?
>
> Below is an example of the log file generated for a map task.
>
> Task
>
> Complete
>
> Status
>
> Start Time
>
> Finish Time
>
> Errors
>
> Counters
>
> task_201009071545_0028_m_000000<http://172.20.101.33:50030/taskdetails.jsp?jobid=job_201009071545_0028&tipid=task_201009071545_0028_m_000000>
>
> 100.00%
>
> Finished processing 30300 records from the input file:
> hdfs://blade33:9000/user/han6/InputFile/5MB.txt
>
> 21-Sep-2010 12:45:17
>
> 21-Sep-2010 12:45:32 (15sec)
>
> 11<http://172.20.101.33:50030/taskstats.jsp?jobid=job_201009071545_0028&tipid=task_201009071545_0028_m_000000>
>
> Thanks,
> Han Dong
> handong32@gmail.com
>