You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gora.apache.org by "Lewis John McGibbney (Created) (JIRA)" <ji...@apache.org> on 2012/01/30 16:24:10 UTC
[jira] [Created] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Avoid HBase MiniCluster restarts to shorten gora-hbase tests
------------------------------------------------------------
Key: GORA-89
URL: https://issues.apache.org/jira/browse/GORA-89
Project: Gora
Issue Type: Improvement
Components: storage-hbase
Affects Versions: 0.2-incubating
Reporter: Lewis John McGibbney
Fix For: 0.2-incubating
Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
Just implement the cluster as a singleton and clean up the tables in
between test by doing a scan and deletes for all rows. It's much
faster than restarting the cluster.
For code referenece please see the implementation here[1]. The class is
HBaseClusterSingleton. It needs some refactoring but I think it's
enough to speed your tests.
Thanks Ioan for the heads up.
[1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Ferdy Galema (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213529#comment-13213529 ]
Ferdy Galema commented on GORA-89:
----------------------------------
Lewis, could you unset "fix version 0.2"? Thanks.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
> Attachments: GORA-89.patch
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199759#comment-13199759 ]
Lewis John McGibbney commented on GORA-89:
------------------------------------------
This looks neat and tidy. Is anyone interested in looking at the this as an alternative. It would be nice to compare test functionality... or time at least.
Thanks Ioan for posting this up.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Lewis John McGibbney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney updated GORA-89:
-------------------------------------
Attachment: GORA-89.patch
This is by no means absolute. It is far from it. I had some spare time so set about it. One of you HBase literate guys would make me very happy if you could pick this patch up and add the finishing touches. :0)
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
> Attachments: GORA-89.patch
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Ferdy Galema (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213527#comment-13213527 ]
Ferdy Galema commented on GORA-89:
----------------------------------
I'm cannot prevent the JVM from hanging on exit. The runtime hook is not executed. Starting the mini cluster in a daemon thread does work either.
Reusing the cluster between tests depends heavily upon HBASE-4448. We need to postpone this issue until we can make use of this improved testing utility. Of course it still may be possible to shorten the duration of the HBase tests in another way (for example by not using full-blown cluster instances every time, but simply the required components for every test.). I will create a separate jira issues for those improvements when I get around to implementing them.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
> Attachments: GORA-89.patch
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Ferdy Galema (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196193#comment-13196193 ]
Ferdy Galema commented on GORA-89:
----------------------------------
This sounds like a good plan. It will rid us of the deprecated HBaseClusterTestCase too. We can fully rely on HBaseTestingUtility, and it seems that this is what HBaseClusterSingleton is using.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2-incubating
> Reporter: Lewis John McGibbney
> Fix For: 0.2-incubating
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Ferdy Galema (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210277#comment-13210277 ]
Ferdy Galema commented on GORA-89:
----------------------------------
I will take a look at it and get back at this.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
> Attachments: GORA-89.patch
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Lewis John McGibbney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney updated GORA-89:
-------------------------------------
Fix Version/s: (was: 0.2)
0.3
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.3
>
> Attachments: GORA-89.patch
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GORA-89) Avoid HBase MiniCluster restarts to
shorten gora-hbase tests
Posted by "Ioan Eugen Stan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GORA-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199647#comment-13199647 ]
Ioan Eugen Stan commented on GORA-89:
-------------------------------------
Just updated the code a little bit. The current implementation should be safer, more singleton like.
The idea is to
- build() the cluster and get an instance.
- clear the tables before every test method ( call clearTables() in @Before annotated method)
It should be pretty clear from the code. Good luck.
> Avoid HBase MiniCluster restarts to shorten gora-hbase tests
> ------------------------------------------------------------
>
> Key: GORA-89
> URL: https://issues.apache.org/jira/browse/GORA-89
> Project: Apache Gora
> Issue Type: Improvement
> Components: storage-hbase
> Affects Versions: 0.2
> Reporter: Lewis John McGibbney
> Fix For: 0.2
>
>
> Currently our hbase tests are taking forever and a day. We should shorten the time by avoiding MiniCluster restarts.
> Just implement the cluster as a singleton and clean up the tables in
> between test by doing a scan and deletes for all rows. It's much
> faster than restarting the cluster.
> For code referenece please see the implementation here[1]. The class is
> HBaseClusterSingleton. It needs some refactoring but I think it's
> enough to speed your tests.
> Thanks Ioan for the heads up.
> [1] http://svn.apache.org/repos/asf/james/mailbox/trunk/hbase/src/test/java/org/apache/james/mailbox/hbase/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira