[jira] Updated: (HADOOP-5916) Standardize fall-back value of test.build.data for testing directories

["Jakob Homan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-5916:
--------------------------------

    Description: 
Currently when the test.build.data property is not set, as happens when run through some configurations of Eclipse, the fall-back value varies wildly.  Most calls set to /tmp, which is not good as it is beyond the scope of the ant clean task and thus will not be deleted.  Others default to "." which can drop the test files right in the current directory.  Speaking with Konstanin, it seems the correct location should be "build/test/data" to ensure any files that are created are within the scope of Ant's clean command.

Attached is the current variation in this setting.

  was:
Currently when the test.build.data property is not set, as happens when run through some configurations of Eclipse, the fall-back value varies wildly.  Most calls set to /tmp, which is not good as it is beyond the scope of the ant clean task and thus will not be deleted.  Others default to "." which can drop the test files right in the current directory.  Speaking with Konstanin, it seems the correct location should be "build/test/data" to ensure any files that are created are within the scope of Ant's clean command.

Current variation of setting the test.build.data:
{noformat}
./contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java:  final static String TEST_DIR = new File(System.getProperty("test.build.data",
./contrib/sqoop/src/java/org/apache/hadoop/sqoop/ImportOptions.java:    String tmpDir = System.getProperty("test.build.data", "/tmp/");
./contrib/sqoop/src/test/org/apache/hadoop/sqoop/testutil/ImportJobTestCase.java:    String tmpDir = System.getProperty("test.build.data", "/tmp/");
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestAutoInputFormat.java:    Path dir = new Path(System.getProperty("test.build.data", ".") + "/mapred");
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleArchiveFiles.java:      "-jobconf", "stream.tmpdir=" + System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleCachefiles.java:          "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestRawBytesStreaming.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamAggregate.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamDataProtocol.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamedMerge.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreaming.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingAsDifferentUser.java:            "stream.tmpdir=" + System.getProperty("test.build.data", "/tmp") };
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingCombiner.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingEmptyInpNonemptyOut.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingExitStatus.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingFailure.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingKeyValue.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingSeparator.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStderr.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamReduceNone.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestSymLink.java:          "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestTypedBytesStreaming.java:      "-jobconf", "stream.tmpdir="+System.getProperty("test.build.data","/tmp"),
./contrib/streaming/src/test/org/apache/hadoop/streaming/TestUlimit.java:                   System.getProperty("test.build.data","/tmp")
./contrib/streaming/src/test/org/apache/hadoop/streaming/UtilTest.java:    antTestDir_ = System.getProperty("test.build.data", userDir_);
./contrib/streaming/src/test/org/apache/hadoop/typedbytes/TestIO.java:    this.tmpfile = new File(System.getProperty("test.build.data", "/tmp"),
./examples/org/apache/hadoop/examples/DBCountPageView.java:        System.getProperty("test.build.data",".") + "/URLAccess");
./test/core/org/apache/hadoop/filecache/TestDistributedCache.java:    new Path(System.getProperty("test.build.data","/tmp/cachebasedir"))
./test/core/org/apache/hadoop/filecache/TestDistributedCache.java:    System.getProperty("test.build.data", "/tmp/distributedcache");
./test/core/org/apache/hadoop/fs/kfs/TestKosmosFileSystem.java:        baseDir = new Path(System.getProperty("test.build.data", "/tmp" ) +
./test/core/org/apache/hadoop/fs/TestChecksumFileSystem.java:    = System.getProperty("test.build.data","build/test/data/work-dir/localfs");
./test/core/org/apache/hadoop/fs/TestDFVariations.java:      super(new File(System.getProperty("test.build.data","/tmp")), 0L);
./test/core/org/apache/hadoop/fs/TestDU.java:      System.getProperty("test.build.data","/tmp"), "dutmp");
./test/core/org/apache/hadoop/fs/TestGetFileBlockLocations.java:      System.getProperty("test.build.data", "/tmp/testGetFileBlockLocations");
./test/core/org/apache/hadoop/fs/TestLocalFileSystem.java:    = System.getProperty("test.build.data","build/test/data/work-dir/localfs");
./test/core/org/apache/hadoop/fs/TestTrash.java:    new Path(new File(System.getProperty("test.build.data","/tmp")
./test/core/org/apache/hadoop/fs/TestTruncatedInputBug.java:    new Path(System.getProperty("test.build.data","/tmp"))
./test/core/org/apache/hadoop/io/TestArrayFile.java:    System.getProperty("test.build.data",".") + "/test.array";
./test/core/org/apache/hadoop/io/TestBloomMapFile.java:    Path dirName = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java:    Path dirName = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java:    Path dirName = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestMapFile.java:    Path dirName = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestSequenceFileSerialization.java:    Path file = new Path(System.getProperty("test.build.data",".") +
./test/core/org/apache/hadoop/io/TestSetFile.java:    System.getProperty("test.build.data",".") + "/test.set";
./test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java:    System.getProperty("test.build.data","build/test/data");
./test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java:    final String SCRIPT_TEST_DIR = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/MiniDFSCluster.java:    return System.getProperty("test.build.data", "build/test/data") + "/dfs/";
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java:      File baseDir = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java:    File baseDir = new File(System.getProperty("test.build.data",
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java:      System.getProperty("test.build.data", "build/test/data"), "dfs/");
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStartup.java:    String baseDir = System.getProperty("test.build.data", "/tmp");
./test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java:    String baseDir = System.getProperty("test.build.data",  "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java:    new Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs/org/apache/hadoop/hdfs/TestDFSStartupVersions.java:                                               System.getProperty("test.build.data","/tmp").toString().replace(' ', '+'));
./test/hdfs/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java:    String dataDir = System.getProperty("test.build.data", "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/TestFileStatus.java:    new Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs/org/apache/hadoop/hdfs/TestFSInputChecker.java:    String dir = System.getProperty("test.build.data", ".");
./test/hdfs/org/apache/hadoop/hdfs/TestHDFSServerPorts.java:    return System.getProperty("test.build.data", "build/test/data");
./test/hdfs/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestDelimitedImageVisitor.java:  private static String ROOT = System.getProperty("test.build.data","/tmp");
./test/hdfs/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:  private static String ROOT = System.getProperty("test.build.data","/tmp");
./test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java:      System.getProperty("test.build.data","/tmp").replace(' ', '+'));
./test/hdfs-with-mr/org/apache/hadoop/fs/DFSCIOTest.java:  private static String TEST_ROOT_DIR = System.getProperty("test.build.data","/benchmarks/DFSCIOTest");
./test/hdfs-with-mr/org/apache/hadoop/fs/DistributedFSCheck.java:  private static Path TEST_ROOT_DIR = new Path(System.getProperty("test.build.data","/benchmarks/DistributedFSCheck"));
./test/hdfs-with-mr/org/apache/hadoop/fs/TestCopyFiles.java:    new Path(System.getProperty("test.build.data","/tmp"))
./test/hdfs-with-mr/org/apache/hadoop/fs/TestDFSIO.java:  private static String TEST_ROOT_DIR = System.getProperty("test.build.data","/benchmarks/TestDFSIO");
./test/hdfs-with-mr/org/apache/hadoop/fs/TestFileSystem.java:  private static String ROOT = System.getProperty("test.build.data","fs_test");
./test/hdfs-with-mr/org/apache/hadoop/io/TestSequenceFileMergeProgress.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/hdfs-with-mr/org/apache/hadoop/tools/TestDistCh.java:    new Path(System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:    Path file = new Path(System.getProperty("test.build.data",".")+"/test.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:      new Path(System.getProperty("test.build.data",".")+"/test.rc.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:      new Path(System.getProperty("test.build.data",".")+"/test.bc.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:    Path file = new Path(System.getProperty("test.build.data",".")+"/test.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:      new Path(System.getProperty("test.build.data",".")+"/test.sorted.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:      new Path(System.getProperty("test.build.data",".")+"/test.rc.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:      new Path(System.getProperty("test.build.data",".")+"/test.bc.seq.metadata");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:    Path path1 = new Path(System.getProperty("test.build.data",".")+"/test1.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:    Path path2 = new Path(System.getProperty("test.build.data",".")+"/test2.seq");
./test/mapred/org/apache/hadoop/io/TestSequenceFile.java:    Path path = new Path(System.getProperty("test.build.data",".")+"/broken.seq");
./test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java:    Path rootDataDir = new Path(System.getProperty("test.build.data", "."), "TestJobControlData");
./test/mapred/org/apache/hadoop/mapred/jobcontrol/TestLocalJobControl.java:    Path rootDataDir = new Path(System.getProperty("test.build.data", "."),
./test/mapred/org/apache/hadoop/mapred/join/TestWrappedRecordReaderClassloader.java:    Path testdir = new Path(System.getProperty("test.build.data", "/tmp"))
./test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java:      String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestChainMapReduce.java:      String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestLineInputFormat.java:    new Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/lib/TestMultipleOutputs.java:      String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/lib/TestMultithreadedMapRunner.java:      String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/MRBench.java:    new Path(System.getProperty("test.build.data","/benchmarks/MRBench"));
./test/mapred/org/apache/hadoop/mapred/MRCaching.java:    String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:      String localPathRoot = System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java:      String localPathRoot = System.getProperty("test.build.data","/tmp")
./test/mapred/org/apache/hadoop/mapred/TestCommandLineJobSubmission.java:  File buildDir = new File(System.getProperty("test.build.data", "/tmp"));
./test/mapred/org/apache/hadoop/mapred/TestEmptyJob.java:      new File(System.getProperty("test.build.data", "/tmp")).toURI()
./test/mapred/org/apache/hadoop/mapred/TestFileInputFormatPathFilter.java:      new Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestFileOutputCommitter.java:     System.getProperty("test.build.data", "."), "output");
./test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java:      String localPathRoot = System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapred/TestIndexCache.java:    Path p = new Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestIndexCache.java:    Path p = new Path(System.getProperty("test.build.data", "/tmp"),
./test/mapred/org/apache/hadoop/mapred/TestJobHistoryParsing.java:    Path historyDir = new Path(System.getProperty("test.build.data", "."), 
./test/mapred/org/apache/hadoop/mapred/TestJobHistoryVersion.java:    new Path(System.getProperty("test.build.data", "/tmp"), 
./test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:    new Path(System.getProperty("test.build.data","/tmp"), "jip-testing");
./test/mapred/org/apache/hadoop/mapred/TestJobQueueInformation.java:    new Path(System.getProperty("test.build.data","/tmp"), 
./test/mapred/org/apache/hadoop/mapred/TestJobTrackerRestart.java:    new Path(System.getProperty("test.build.data","/tmp"), 
./test/mapred/org/apache/hadoop/mapred/TestJobTrackerSafeMode.java:    new Path(System.getProperty("test.build.data", "/tmp"), "jt-safemode");
./test/mapred/org/apache/hadoop/mapred/TestKeyValueTextInputFormat.java:    new Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestMapRed.java:        System.getProperty("test.build.data","/tmp")).makeQualified(fs);
./test/mapred/org/apache/hadoop/mapred/TestMapRed.java:        System.getProperty("test.build.data","/tmp")).makeQualified(fs);
./test/mapred/org/apache/hadoop/mapred/TestMiniMRLocalFS.java:    new File(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java:    String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestMiniMRTaskTempDir.java:    String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestMiniMRWithDFS.java:        new File(System.getProperty("test.build.data","/tmp"))
./test/mapred/org/apache/hadoop/mapred/TestMultiFileInputFormat.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestMultipleTextOutputFormat.java:                      new Path(System.getProperty("test.build.data", "."), 
./test/mapred/org/apache/hadoop/mapred/TestRecoveryManager.java:    new Path(System.getProperty("test.build.data", "/tmp"), 
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryInputFormat.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:      new Path(new Path(new Path(System.getProperty("test.build.data",".")), 
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsBinaryOutputFormat.java:    Path outputdir = new Path(System.getProperty("test.build.data",".") 
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileAsTextInputFormat.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileInputFilter.java:  private static final Path inDir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSequenceFileInputFormat.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
./test/mapred/org/apache/hadoop/mapred/TestSpilledRecordsCounter.java:    String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestTaskFail.java:    String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data",
./test/mapred/org/apache/hadoop/mapred/TestTextInputFormat.java:    new Path(new Path(System.getProperty("test.build.data", "."), "data"),
./test/mapred/org/apache/hadoop/mapred/TestTextOutputFormat.java:                      new Path(System.getProperty("test.build.data", "."), 
./test/mapred/org/apache/hadoop/mapred/ThreadedMapBenchmark.java:    new Path(System.getProperty("test.build.data", 
./test/mapred/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java:    final Path testdir = new Path(System.getProperty("test.build.data", "/tmp")
./test/mapred/org/apache/hadoop/mapreduce/TestMapReduceLocal.java:    new Path(System.getProperty("test.build.data","/tmp"));
./test/mapred/org/apache/hadoop/mapreduce/TestValueIterReset.java:    new Path(System.getProperty("test.build.data","/tmp"));
./test/mapred/org/apache/hadoop/record/TestRecordWritable.java:    Path dir = new Path(System.getProperty("test.build.data",".") + "/mapred");
{noformat}


Deleted overly long list of offending statements.  Will attach as a file.

> Standardize fall-back value of test.build.data for testing directories
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-5916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5916
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>         Attachments: test.build.data.txt
>
>
> Currently when the test.build.data property is not set, as happens when run through some configurations of Eclipse, the fall-back value varies wildly.  Most calls set to /tmp, which is not good as it is beyond the scope of the ant clean task and thus will not be deleted.  Others default to "." which can drop the test files right in the current directory.  Speaking with Konstanin, it seems the correct location should be "build/test/data" to ensure any files that are created are within the scope of Ant's clean command.
> Attached is the current variation in this setting.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.