ptest tmp directories and test flakiness

[Sergey Shelukhin <se...@hortonworks.com>]

Hi.

We have many test failures due to flakiness on ptest machines; looks like
tmp directory is deleted while tests are running:

2018-05-18T10:24:44,991 WARN [Thread-3915] mapred.LocalJobRunner:
job_local632888732_0106
java.io.FileNotFoundException: File
file:/tmp/hadoop/mapred/staging/hiveptest632888732/.staging/job_local632888
732_0106/job.splitmetainfo does not exist
…

Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input path does not exist: file:/tmp/temp1540619121/tmp-2080326801

…
etc.

1) Can we have tmp directory NOT cleaned up while tests are running? I
wonder if it’s easy to nuke between runs.

2) Otherwise we need to weed out all the tests that use tmp and make them
not use it. I’m not sure about the best way to do this… hadoop/mapred
seems to come from mapreduce.jobtracker.staging.root.dir and
hadoop.tmp.dir, but at least after some time looking I cannot find where
we set hadoop.tmp.dir to /tmp/hadoop, and it also doesn’t match the
default value that has username.
Where the other one comes I’m not sure at all.
I wonder if it’s viable to deny ptest user access to tmp temporarily, then
see what fails at the earliest possible point?