Problem scaling working hadoop job

[Steve Lewis <lo...@gmail.com>]

I have a Hadoop job that I have successfully run with an input set of about
50 million input  records. To test scaling and prepare for where we plan to
be a year or two from now I tried the same job with a about 4 times as many
records

most map tasks fail with the message
could not find any valid local directory for
tasktracker/jobcache/job.../jars

The first  job is writing about 4 TB and is running on an 0.23 cluster.

My general understanding is that this message occurs when a tmp directory
on local drive gets full.

I have requested that systems restart the cluster.

My questions are:
1) Are there commands to run on a slave to see the issue?
2) Will restarting the cluster clear things out and help?
3) Are there ways to tune the job to mitigate this issue?