You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by Arwin Tio <ar...@hotmail.com> on 2018/11/30 20:18:07 UTC

Is mapreduce.job.local.dir data lost if a machine dies

When a machine dies in a Hadoop cluster, the data in mapreduce.job.local.dir is presumably lost. This data contains mapper output files as well as reducer input files. In addition, data stored in the Shuffle Fetcher memory is lost.

How do we ensure that this lost data is re-processed when a machine dies?

Thanks,

Arwin