You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nkeywal (Created) (JIRA)" <ji...@apache.org> on 2011/10/20 17:43:11 UTC
[jira] [Created] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
"test.build.data" property overused leading to write data at the wrong place
----------------------------------------------------------------------------
Key: HBASE-4634
URL: https://issues.apache.org/jira/browse/HBASE-4634
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
"test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
It seems that this value is already used by MiniDFS (i.e. outside of HBase):
"Name is as it is because mini dfs has hard-codings to put test data here."
As it is today, there is at least a bug in HBaseTestingUtility:
{noformat}
public void initTestDir() {
if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
clusterTestBuildDir = setupClusterTestBuildDir();
System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
}
}
{noformat}
if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131912#comment-13131912 ]
nkeywal commented on HBASE-4634:
--------------------------------
Ok, I've implemented something, currently under testing.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136701#comment-13136701 ]
Hudson commented on HBASE-4634:
-------------------------------
Integrated in HBase-TRUNK #2372 (See [https://builds.apache.org/job/HBase-TRUNK/2372/])
HBASE-4634 'test.build.data' property overused leading to write data at the wrong place
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.92.0
>
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch, 20111024_4634_all.v5.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133045#comment-13133045 ]
nkeywal commented on HBASE-4634:
--------------------------------
I am going to attach a second version of the patch. I checked indent, it seems ok to me.
It took more time than expected because I haven't managed to escape some modifications to the get files functions. Not much impacts for the user. Basically, there is now a clear distinction between:
- base directory: configurable directory where all tests put their data
- data directory: sub directory used by the test
- cluster directory: subdirectory for the cluster stuff (dfs or hbase)
This will help to have multiple HBaseTestingUtility in the same JVM (// or not).
I have also removed a duplicate getter (something as getHbaseCluster while getHBase cluster exists).
It includes the fixes for HBASE-4639 and most if not all the bad usage mentioned in HBASE-4326 (but I did not fix the root issue of HBASE-4326).
It seems to work on my env, currently n-checking again.
I have a systematic errors in the test, but it's in #org.apache.hadoop.hbase.io.hfile.TestHFileBlock#testBlockHeapSize , I believe it was there at the beginning.
I have random errors in: TestDistributedLogSplitting, regionserver.TestServerCustomProtocol, client.TestAdmin master.TestHMasterRPCException, coprocessor.TestRegionServerCoprocessorExceptionWithAbort, wal.TestLogRolling, CatalogTrackerOnCluster and a few else, but it works when I replay the test set, so I tend to think they belong to the flaky test sets.
At the end of the day, the patch is not that medium... Sorry for that.
Will add a new comment when the tests will be over and I will have rechecked the flaky one, but you can already have a look.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4634:
---------------------------
Attachment: 20111023_4634_all.v4.patch
Tests ok...
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132133#comment-13132133 ]
nkeywal commented on HBASE-4634:
--------------------------------
I found a bug, so there is another testing iteration. setting TEST_DIRECTORY_KEY as private worked.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4634:
---------------------------
Attachment: 20111021_4634_all.v2.patch
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132003#comment-13132003 ]
nkeywal commented on HBASE-4634:
--------------------------------
For TestStoreFile, It would need to be applied here, even it's doing more
than just removing the "static dependency on the TestingTool" : it includes
4639.
It will take into account your comments and publish an updated version.
- with BASE_TEST_DIRECTORY
- private for TEST_DIRECTORY_KEY should to it, I will try.
+ indent stuff
On Thu, Oct 20, 2011 at 10:25 PM, stack (Commented) (JIRA)
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133048#comment-13133048 ]
Ted Yu commented on HBASE-4634:
-------------------------------
TestHFileBlock#testBlockHeapSize is fixed by addendum to HBASE-4219
Please refresh your workspace.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131987#comment-13131987 ]
stack commented on HBASE-4634:
------------------------------
Please use the same indent as the surrounding file (two spaces for a tab).
Should we deprecate ' public static final String TEST_DIRECTORY_KEY = "test.build.data";' so we can later change it to private or protected access?
'root' is a pretty overloaded term in these parts. There is unix root and then there is hbase root region and hbase.rootdir. You are using ROOT_TEST_DIRECTORY in place of TEST_DIRECTORY. Maybe BASE_TEST_DIRECTORY instead?
FYI, the surrounding code has spaces between operators. e.g. '+ if (clusterTestBuildDir==null){' should be '+ if (clusterTestBuildDir == null) {'
Otherwise, patch is going in right direction IMO. Good stuff.
I'm not sure what to do with the patch 20111020_4639_TestStoreFile.patch. Is it to be applied here or over in hbase-4639?
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4634.
--------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: Reviewed
Committed 0.92 and trunk. Thank you for the patch N.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.92.0
>
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch, 20111024_4634_all.v5.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4634:
---------------------------
Attachment: 20111022_4634_all.v3.patch
A few more modifications in some tests. All test runs fine.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4634:
---------------------------
Attachment: 20111024_4634_all.v5.patch
v5, all tests ok, after a refresh.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch, 20111024_4634_all.v5.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nkeywal updated HBASE-4634:
---------------------------
Attachment: 20111020_4639_TestStoreFile.patch
2011020_4634_all.patch
The fix, currently under testing on the trunk. I attach right now to get comments sooner, at it impacts multiple files.
The function to get the test directory is not static anymore, that's why many files are impacted.
It's cleaner, and will facilitate running the tests in the same JVM a day.
TestStoreFile includes the fix for HBASE-4639, there is an overlap here.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134171#comment-13134171 ]
stack commented on HBASE-4634:
------------------------------
@nkeywal Mind refreshing your TRUNK? Patch did not apply cleanly and I got this when trying to build it:
{code}
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[265,8] cannot find symbol
symbol: variable ShowTime
ShowTime.start();
/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[270,8] cannot find symbol
symbol: variable ShowTime
ShowTime.pub("public void run() ");
/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[401,4] cannot find symbol
symbol : variable ShowTime
location: class org.apache.hadoop.hbase.master.TestDistributedLogSplitting
/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:[403,4] cannot find symbol
symbol : variable ShowTime
location: class org.apache.hadoop.hbase.master.TestDistributedLogSplitting
/Users/stack/checkout/clean-trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java:[228,39] cannot find symbol
symbol : method getTestDir()
location: class org.apache.hadoop.hbase.HBaseTestingUtility
{code}
Thanks boss.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133415#comment-13133415 ]
nkeywal commented on HBASE-4634:
--------------------------------
v4 coming after:
- some impacts of shutdown throwing Exception instead of IOException were not taken into account
- startZK before starting the cluster was puting the file in the current directory (old bug I think).
Full test in progress.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136626#comment-13136626 ]
Hudson commented on HBASE-4634:
-------------------------------
Integrated in HBase-0.92 #83 (See [https://builds.apache.org/job/HBase-0.92/83/])
HBASE-4634 'test.build.data' property overused leading to write data at the wrong place
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestCloseRegionHandler.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationPeer.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestHQuorumPeer.java
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Fix For: 0.92.0
>
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch, 20111024_4634_all.v5.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "nkeywal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134181#comment-13134181 ]
nkeywal commented on HBASE-4634:
--------------------------------
that's an error I missed in my patch. Ok to refresh.
On Mon, Oct 24, 2011 at 5:51 PM, stack (Commented) (JIRA)
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
> Attachments: 2011020_4634_all.patch, 20111020_4639_TestStoreFile.patch, 20111021_4634_all.v2.patch, 20111022_4634_all.v3.patch, 20111023_4634_all.v4.patch
>
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) "test.build.data" property overused
leading to write data at the wrong place
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131722#comment-13131722 ]
Ted Yu commented on HBASE-4634:
-------------------------------
>From MiniDFSCluster.java:
{code}
public static String getBaseDirectory() {
return System.getProperty(PROP_TEST_BUILD_DATA, "build/test/data") + "/dfs/";
}
{code}
I think we can introduce a new key for the test data used by hbase.
> "test.build.data" property overused leading to write data at the wrong place
> ----------------------------------------------------------------------------
>
> Key: HBASE-4634
> URL: https://issues.apache.org/jira/browse/HBASE-4634
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.92.0
> Environment: all
> Reporter: nkeywal
> Assignee: nkeywal
>
> "test.build.data" is overloaded in HBase.At the beginning, it's the "Default parent directory for test output.", but then it's rewritten to be the directory itself in functions like HBaseTestingUtility#startMiniDFSCluster
> It seems that this value is already used by MiniDFS (i.e. outside of HBase):
> "Name is as it is because mini dfs has hard-codings to put test data here."
> As it is today, there is at least a bug in HBaseTestingUtility:
> {noformat}
> public void initTestDir() {
> if (System.getProperty(TEST_DIRECTORY_KEY) == null) {
> clusterTestBuildDir = setupClusterTestBuildDir();
> System.setProperty(TEST_DIRECTORY_KEY, clusterTestBuildDir.getPath());
> }
> }
> {noformat}
> if you set a value for "test.build.data", the test dir will be the parent directory and not a temp subdir, leading to issues as multiple tests will end-ups in the same (bad) directory. This function is barely used today, hence it's not visible, but I would like to use it in some new code.
> A possible fix is to remove the check for null and continue with the overloading, but I don't think it would be a big issue to create a new key(like "test.build.data.rootdirectory") specific to the root directory and to use "test.build.data" only to communicate with MiniDFS. Feedback welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira