Questions about mapred.local.dir

[Kevin Tse <ke...@gmail.com>]

Hi, everyone.
I am running a Hadoop-0.19.2 cluster of 4 linux boxes. it is bad that for
the moment we don't have much available disk space on these 4 nodes, 1.5 TB
in total, spreading over 8 disks, 2 for each node. and the available space
in these disks are not equal.

I have the following configuration.
I tried to run a job that would write estimated amount of 1.2
TB intermediate data. I thought that with the following configuration, no
any disk's space would drop under 2 GB, but it was not ture, two of the
disks' space dropped to 0 before the job finished, so I had to kill the job
and free the space for other applications running on those machines.

we are going to buy 3*1TB disks, this may solve the problem, but I still
want to know how to properly set the following 3 properties.

<property>
      <name>mapred.local.dir</name>
      <value>/disk1/mapred/local,/disk2/mapred/local</value>
   </property>
<property>
  <name>mapred.local.dir.minspacestart</name>
  <value>4096000000</value>
</property>
<property>
  <name>mapred.local.dir.minspacekill</name>
  <value>2048000000</value>
</property>

And there's another problem, while my MR job is being executed in the hadoop
cluster, for each tasktacker, there are many INFO log message as this:

2010-06-07 16:51:20,721 INFO org.apache.hadoop.mapred.TaskTracker:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201006071648_0002
/attempt_201006071648_0002_r_000001_0/output/file.out
in any of the configured local directories

I don't know whether this is harmless, but it seems so cause my MR job
completed successfully.

And another question, is it possible to make the reduces start to run before
all the maps complete?

Thank you in advance.
Kevin Tse