You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2008/03/07 02:24:57 UTC
[jira] Created: (HBASE-498) Use an external DFS instead of spinning
up a MiniDFS for every test
Use an external DFS instead of spinning up a MiniDFS for every test
-------------------------------------------------------------------
Key: HBASE-498
URL: https://issues.apache.org/jira/browse/HBASE-498
Project: Hadoop HBase
Issue Type: Sub-task
Reporter: Bryan Duxbury
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-498) Use an external DFS instead of spinning
up a MiniDFS for every test
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-498:
--------------------------------
Fix Version/s: (was: 0.2.0)
Affects Version/s: 0.2.0
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Affects Versions: 0.2.0
> Reporter: Bryan Duxbury
> Priority: Minor
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-498) Use an external DFS instead of
spinning up a MiniDFS for every test
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576356#action_12576356 ]
stack commented on HBASE-498:
-----------------------------
Great. We can call this at end of test? Shall we start in on a refactor of tests that cuts out spin up of minidfs and that instead just reads the filesystem to use from $HBASE_HOME/src/test/hbase-site.xml? (On hudson can fill this file w/ the hdfs to use)
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Bryan Duxbury
> Fix For: 0.2.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-498) Use an external DFS instead of
spinning up a MiniDFS for every test
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576287#action_12576287 ]
stack commented on HBASE-498:
-----------------------------
I like the idea of using the configured filesystem and defaulting local filesystem (if tests on windows pass on local filesystem).
Jim, did you ever get chance at figuring why recursive delete is not working when on local filesystem on windows though seems to work when you do ./bin/hadoop -rmr /SOME_DIR_WITH_SUBDIRS on your windows machine?
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Bryan Duxbury
> Fix For: 0.2.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-498) Use an external DFS instead of spinning
up a MiniDFS for every test
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-498:
--------------------------------
Component/s: test
Fix Version/s: 0.2.0
Description:
My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Bryan Duxbury
> Fix For: 0.2.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-498) Use an external DFS instead of spinning
up a MiniDFS for every test
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-498:
--------------------------------
Priority: Minor (was: Major)
Sub-issue was more important that parent issue. Correcting.
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-498) Use an external DFS instead of
spinning up a MiniDFS for every test
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576352#action_12576352 ]
Jim Kellerman commented on HBASE-498:
-------------------------------------
FsShell has a special version of delete that works recursively. FileSystem.delete is not recursive. It works on unix because you can remove a directory if it is not empty. No so on Windows.
However, when we delete directories, we might want to use FileUtil.fullyDelete instead of FileSystem.delete and that should solve the problem.
> Use an external DFS instead of spinning up a MiniDFS for every test
> -------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Reporter: Bryan Duxbury
> Fix For: 0.2.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-498) [testing] Use an external DFS instead
of spinning up a MiniDFS for every test
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-498.
-------------------------
Resolution: Duplicate
Running a dfscluster for length of test is what hbase-410 is about. Marking this closed as duplicate.
> [testing] Use an external DFS instead of spinning up a MiniDFS for every test
> -----------------------------------------------------------------------------
>
> Key: HBASE-498
> URL: https://issues.apache.org/jira/browse/HBASE-498
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: test
> Affects Versions: 0.2.0
> Reporter: Bryan Duxbury
> Priority: Minor
> Fix For: 0.21.0
>
>
> My testing suggests that we could shave about a minute off the entire suite if we only had to start up a single DFS instance instead of a new one for each test. Stack has investigated using ant to perform this initialization task, but come up empty handed.
> I propose that we should expect the developer running the test suite to start a DFS before running "ant test". We could provide a bin/hbase command that does specifically that in order to make it easy. This would save not only the initial start up time but all the startup time of all tests and test suite runs you execute in a session of development and testing.
> Do people think that an extra step to testing would be too hard for new (or lazy existing) developers?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.