_temporary directory getting deleted mid-job?

[Aaron Kimball <aa...@cloudera.com>]

I saw some puzzling behavior tonight when running a MapReduce program I
wrote.

It would perform the mapping just fine, and would begin to shuffle. It got
to 33% complete reduce (end of shuffle) and then the task fails, claiming
that <output_dir>/_temporary was deleted.

I didn't touch HDFS while this was going on.

I tried running the job multiple more times, and this repeated twice more.
Puzzlingly, I was doing bin/hadoop fs -ls <output_dir> periodically in
another window. The _temporary directory got created just fine, but at some
point after shuffling began, it was removed.

I tried to see if I could manually race this, so I did a mkdir _temporary,
and the job proceeded just fine. Even more bizarre, the removal of the
_temporary directory did not occur on any subsequent MR jobs (executions of
the same, unmodified program). So I can't reproduce the bug.

This is on 0.18.2.

It went away, so I'm not *too* concerned, but I'd rather not deal with
heisenbugs if at all possible

So: has anyone seen this behavior? Have you figured out how to reproduce it,
or even better, prevent it?

Thanks,
- Aaron