Identify the input file for a failed mapper/reducer

[Jason Fennell <jd...@gmail.com>]

Is there a way to identify the input file a mapper was running on when
it failed?  When a large job fails because of bad input lines I have
to resort to rerunning the entire job to isolate a single bad line
(since the log doesn't contain information on the file that that
mapper was running on).

Basically, I would like to be able to do one of the following:
1. Find the file that a mapper was running on when it failed
2. Find the block that a mapper was running on when it failed (and be
able to find file names from block ids)

I haven't been able to find any documentation on facilities to
accomplish either (1) or (2), so I'm hoping someone on this list will
have a suggestion.

I am using the Hadoop streaming API on hadoop 0.18.2.

-Jason

Re: Identify the input file for a failed mapper/reducer

[Rasit OZDAS <ra...@gmail.com>]

Two quotes for this problem:

"Streaming map tasks should have a "map_input_file" environment
variable like the following:
map_input_file=hdfs://HOST/path/to/file"

"the value for map.input.file gives you the exact information you need."

(didn't try)
Rasit

2009/3/26 Jason Fennell <jd...@gmail.com>:
> Is there a way to identify the input file a mapper was running on when
> it failed?  When a large job fails because of bad input lines I have
> to resort to rerunning the entire job to isolate a single bad line
> (since the log doesn't contain information on the file that that
> mapper was running on).
>
> Basically, I would like to be able to do one of the following:
> 1. Find the file that a mapper was running on when it failed
> 2. Find the block that a mapper was running on when it failed (and be
> able to find file names from block ids)
>
> I haven't been able to find any documentation on facilities to
> accomplish either (1) or (2), so I'm hoping someone on this list will
> have a suggestion.
>
> I am using the Hadoop streaming API on hadoop 0.18.2.
>
> -Jason
>



-- 
M. Raşit ÖZDAŞ