You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mark Laffoon <ml...@semanticresearch.com> on 2011/01/18 23:57:05 UTC
hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Sorry about the goofy subject but this is driving me nuts. Any answers to
any of these questions would be greatly appreciated (well, except a
generic "yes").
My goal is to be able to run a sequence of unit tests (well, integration
tests really) on our persistence stack. We're using HBaseClusterTestCase.
We have about 25 sets of 20-40 tests that we want to run. Each set needs
to stand up a fresh, independent store. We're using maven to do our
builds.
1. In our build machine environment (Ubuntu, Anthill) and my build
environment (Windows, cygwin), it doesn't work because the test cases
don't tear down properly, and the test folders stick around, messing up
the subsequent tests.
2. In my IDE (IntelliJ on Windows using IDE's internal maven), the
tests don't work because MiniDFSCluster eventually calls "du" which
doesn't exist.
3. Anybody have sample code for setting up a simulated data node? Is
there such thing as a simulated persistence for zookeeper? (Our test data
isn't very large so this would be fine)
One final question that may at least help my blood pressure: has hbase
unit testing improved at all in version 0.90?
Thanks,
Mark
RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Posted by Mark Laffoon <ml...@semanticresearch.com>.
In case anybody stumbles across this thread, I did eventually find the
problem: mvn isn't launching new JVMs so any external resources (or
statics) are retained. Switching to HBaseTestingUtility helped because it
uses a random data folder. We needed to do something similar with another
(non-hadoop) data folder our system creates.
All the test suites are working great now, running back-to-back with no
problems (of course, I've only tried it on 2 of our 5 dev environments,
but it's a start). Now if we could just reduce the setup/teardown time
(it's about 35 seconds on my system).
Thanks
-----Original Message-----
From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
Sent: Friday, January 21, 2011 9:38 AM
To: user@hbase.apache.org
Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
Thanks Jonathan,
Before I file the bug, I'll post here in case it's me being stupid and not
worthy of a Jira. I simplified my test and there are some log entries that
may or may not be misleading. Note at the end of the log the test folder
can't be deleted. I've included the base test class; the real test extends
this and adds a simple noop @Test. hbase-site-test is a copy of the one
yanked from the test jar, with the zookeeper port set to 21810 to match
the hardcoded value.
Mark
-----Original Message-----
From: Jonathan Gray [mailto:jgray@fb.com]
Sent: Thursday, January 20, 2011 4:57 PM
To: user@hbase.apache.org
Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
Sounds like a bug. Want to file a JIRA with as much detail as you can
including log snippets?
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Thursday, January 20, 2011 2:43 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ
> -
arg!
>
> Follow up on this, hopefully Jonathan and/or Stack are still watching
;)
>
> I switched to HBaseTestingUtility (we had a class which was doing
more-or-
> less the same thing with HBaseClusterTestCase). I dealt with my IDE
issues by
> jamming cygwin/bin into my Windows PATH <sigh>. However, we still
> can't run multiple test suites using command-line maven. When the test
> suite finishes running and calls HTU.shutdownMiniCluster, it logs a
> warning
that it
> can't delete the temporary data folder. Looking in the folder, the
zookeeper
> subfolder is being left around. This implies that there is a zookeeper
process
> that isn't letting go. When the next test suite tries to run, it bumps
into port
> issues with the previous process.
> Or something.
>
> Anybody out there doing anything like this?
>
> Is my only solution to run *all* my persist tests in one giant test
suite?
> I really hope not.'
>
> Thanks,
> Mark
>
>
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@fb.com]
> Sent: Tuesday, January 18, 2011 7:56 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ
> -
arg!
>
> Hey Mark. Sorry to hear about your troubles.
>
> There is a new testing facility that has replaced HBaseClusterTestCase.
> Check out HBaseTestingUtility. It's JUnit4 based. One sample usage
> of
it is
> the test TestFromClientSide.
>
> This new one includes support for multiple DataNodes, RegionServers,
> and Masters. Hopefully that will work better for you.
>
> As far as working in your IDE, du is part of Cygwin. So if it's
> saying
not found,
> something in your environment (like your path) is not configured
properly for
> Cygwin and is not finding du.
>
> Hope that helps.
>
> JG
>
> > -----Original Message-----
> > From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> > Sent: Tuesday, January 18, 2011 2:57 PM
> > To: user@hbase.apache.org
> > Subject: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
> arg!
> >
> > Sorry about the goofy subject but this is driving me nuts. Any
> > answers
> to any
> > of these questions would be greatly appreciated (well, except a
> > generic "yes").
> >
> >
> >
> > My goal is to be able to run a sequence of unit tests (well,
> > integration
> tests
> > really) on our persistence stack. We're using HBaseClusterTestCase.
> > We have about 25 sets of 20-40 tests that we want to run. Each set
> > needs
> to
> > stand up a fresh, independent store. We're using maven to do our
builds.
> >
> > 1. In our build machine environment (Ubuntu, Anthill) and my
build
> > environment (Windows, cygwin), it doesn't work because the test
> > cases don't tear down properly, and the test folders stick around,
> > messing up
> the
> > subsequent tests.
> >
> > 2. In my IDE (IntelliJ on Windows using IDE's internal maven),
the
> > tests don't work because MiniDFSCluster eventually calls "du" which
> doesn't
> > exist.
> >
> > 3. Anybody have sample code for setting up a simulated data node?
> Is
> > there such thing as a simulated persistence for zookeeper? (Our test
> data
> > isn't very large so this would be fine)
> >
> >
> >
> > One final question that may at least help my blood pressure: has
> > hbase
> unit
> > testing improved at all in version 0.90?
> >
> >
> >
> > Thanks,
> >
> > Mark
RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Posted by Mark Laffoon <ml...@semanticresearch.com>.
Thanks Jonathan,
Before I file the bug, I'll post here in case it's me being stupid and not
worthy of a Jira. I simplified my test and there are some log entries that
may or may not be misleading. Note at the end of the log the test folder
can't be deleted. I've included the base test class; the real test extends
this and adds a simple noop @Test. hbase-site-test is a copy of the one
yanked from the test jar, with the zookeeper port set to 21810 to match
the hardcoded value.
Mark
-----Original Message-----
From: Jonathan Gray [mailto:jgray@fb.com]
Sent: Thursday, January 20, 2011 4:57 PM
To: user@hbase.apache.org
Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
Sounds like a bug. Want to file a JIRA with as much detail as you can
including log snippets?
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Thursday, January 20, 2011 2:43 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
>
> Follow up on this, hopefully Jonathan and/or Stack are still watching
;)
>
> I switched to HBaseTestingUtility (we had a class which was doing
more-or-
> less the same thing with HBaseClusterTestCase). I dealt with my IDE
issues by
> jamming cygwin/bin into my Windows PATH <sigh>. However, we still can't
> run multiple test suites using command-line maven. When the test suite
> finishes running and calls HTU.shutdownMiniCluster, it logs a warning
that it
> can't delete the temporary data folder. Looking in the folder, the
zookeeper
> subfolder is being left around. This implies that there is a zookeeper
process
> that isn't letting go. When the next test suite tries to run, it bumps
into port
> issues with the previous process.
> Or something.
>
> Anybody out there doing anything like this?
>
> Is my only solution to run *all* my persist tests in one giant test
suite?
> I really hope not.'
>
> Thanks,
> Mark
>
>
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@fb.com]
> Sent: Tuesday, January 18, 2011 7:56 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
>
> Hey Mark. Sorry to hear about your troubles.
>
> There is a new testing facility that has replaced HBaseClusterTestCase.
> Check out HBaseTestingUtility. It's JUnit4 based. One sample usage of
it is
> the test TestFromClientSide.
>
> This new one includes support for multiple DataNodes, RegionServers, and
> Masters. Hopefully that will work better for you.
>
> As far as working in your IDE, du is part of Cygwin. So if it's saying
not found,
> something in your environment (like your path) is not configured
properly for
> Cygwin and is not finding du.
>
> Hope that helps.
>
> JG
>
> > -----Original Message-----
> > From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> > Sent: Tuesday, January 18, 2011 2:57 PM
> > To: user@hbase.apache.org
> > Subject: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
> arg!
> >
> > Sorry about the goofy subject but this is driving me nuts. Any answers
> to any
> > of these questions would be greatly appreciated (well, except a
> > generic "yes").
> >
> >
> >
> > My goal is to be able to run a sequence of unit tests (well,
> > integration
> tests
> > really) on our persistence stack. We're using HBaseClusterTestCase.
> > We have about 25 sets of 20-40 tests that we want to run. Each set
> > needs
> to
> > stand up a fresh, independent store. We're using maven to do our
builds.
> >
> > 1. In our build machine environment (Ubuntu, Anthill) and my
build
> > environment (Windows, cygwin), it doesn't work because the test cases
> > don't tear down properly, and the test folders stick around, messing
> > up
> the
> > subsequent tests.
> >
> > 2. In my IDE (IntelliJ on Windows using IDE's internal maven),
the
> > tests don't work because MiniDFSCluster eventually calls "du" which
> doesn't
> > exist.
> >
> > 3. Anybody have sample code for setting up a simulated data node?
> Is
> > there such thing as a simulated persistence for zookeeper? (Our test
> data
> > isn't very large so this would be fine)
> >
> >
> >
> > One final question that may at least help my blood pressure: has hbase
> unit
> > testing improved at all in version 0.90?
> >
> >
> >
> > Thanks,
> >
> > Mark
RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Posted by Jonathan Gray <jg...@fb.com>.
Sounds like a bug. Want to file a JIRA with as much detail as you can including log snippets?
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Thursday, January 20, 2011 2:43 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
>
> Follow up on this, hopefully Jonathan and/or Stack are still watching ;)
>
> I switched to HBaseTestingUtility (we had a class which was doing more-or-
> less the same thing with HBaseClusterTestCase). I dealt with my IDE issues by
> jamming cygwin/bin into my Windows PATH <sigh>. However, we still can't
> run multiple test suites using command-line maven. When the test suite
> finishes running and calls HTU.shutdownMiniCluster, it logs a warning that it
> can't delete the temporary data folder. Looking in the folder, the zookeeper
> subfolder is being left around. This implies that there is a zookeeper process
> that isn't letting go. When the next test suite tries to run, it bumps into port
> issues with the previous process.
> Or something.
>
> Anybody out there doing anything like this?
>
> Is my only solution to run *all* my persist tests in one giant test suite?
> I really hope not.'
>
> Thanks,
> Mark
>
>
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@fb.com]
> Sent: Tuesday, January 18, 2011 7:56 PM
> To: user@hbase.apache.org
> Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
>
> Hey Mark. Sorry to hear about your troubles.
>
> There is a new testing facility that has replaced HBaseClusterTestCase.
> Check out HBaseTestingUtility. It's JUnit4 based. One sample usage of it is
> the test TestFromClientSide.
>
> This new one includes support for multiple DataNodes, RegionServers, and
> Masters. Hopefully that will work better for you.
>
> As far as working in your IDE, du is part of Cygwin. So if it's saying not found,
> something in your environment (like your path) is not configured properly for
> Cygwin and is not finding du.
>
> Hope that helps.
>
> JG
>
> > -----Original Message-----
> > From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> > Sent: Tuesday, January 18, 2011 2:57 PM
> > To: user@hbase.apache.org
> > Subject: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
> arg!
> >
> > Sorry about the goofy subject but this is driving me nuts. Any answers
> to any
> > of these questions would be greatly appreciated (well, except a
> > generic "yes").
> >
> >
> >
> > My goal is to be able to run a sequence of unit tests (well,
> > integration
> tests
> > really) on our persistence stack. We're using HBaseClusterTestCase.
> > We have about 25 sets of 20-40 tests that we want to run. Each set
> > needs
> to
> > stand up a fresh, independent store. We're using maven to do our builds.
> >
> > 1. In our build machine environment (Ubuntu, Anthill) and my build
> > environment (Windows, cygwin), it doesn't work because the test cases
> > don't tear down properly, and the test folders stick around, messing
> > up
> the
> > subsequent tests.
> >
> > 2. In my IDE (IntelliJ on Windows using IDE's internal maven), the
> > tests don't work because MiniDFSCluster eventually calls "du" which
> doesn't
> > exist.
> >
> > 3. Anybody have sample code for setting up a simulated data node?
> Is
> > there such thing as a simulated persistence for zookeeper? (Our test
> data
> > isn't very large so this would be fine)
> >
> >
> >
> > One final question that may at least help my blood pressure: has hbase
> unit
> > testing improved at all in version 0.90?
> >
> >
> >
> > Thanks,
> >
> > Mark
RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Posted by Mark Laffoon <ml...@semanticresearch.com>.
Follow up on this, hopefully Jonathan and/or Stack are still watching ;)
I switched to HBaseTestingUtility (we had a class which was doing
more-or-less the same thing with HBaseClusterTestCase). I dealt with my
IDE issues by jamming cygwin/bin into my Windows PATH <sigh>. However, we
still can't run multiple test suites using command-line maven. When the
test suite finishes running and calls HTU.shutdownMiniCluster, it logs a
warning that it can't delete the temporary data folder. Looking in the
folder, the zookeeper subfolder is being left around. This implies that
there is a zookeeper process that isn't letting go. When the next test
suite tries to run, it bumps into port issues with the previous process.
Or something.
Anybody out there doing anything like this?
Is my only solution to run *all* my persist tests in one giant test suite?
I really hope not.'
Thanks,
Mark
-----Original Message-----
From: Jonathan Gray [mailto:jgray@fb.com]
Sent: Tuesday, January 18, 2011 7:56 PM
To: user@hbase.apache.org
Subject: RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
Hey Mark. Sorry to hear about your troubles.
There is a new testing facility that has replaced HBaseClusterTestCase.
Check out HBaseTestingUtility. It's JUnit4 based. One sample usage of it
is the test TestFromClientSide.
This new one includes support for multiple DataNodes, RegionServers, and
Masters. Hopefully that will work better for you.
As far as working in your IDE, du is part of Cygwin. So if it's saying
not found, something in your environment (like your path) is not
configured properly for Cygwin and is not finding du.
Hope that helps.
JG
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Tuesday, January 18, 2011 2:57 PM
> To: user@hbase.apache.org
> Subject: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ -
arg!
>
> Sorry about the goofy subject but this is driving me nuts. Any answers
to any
> of these questions would be greatly appreciated (well, except a generic
> "yes").
>
>
>
> My goal is to be able to run a sequence of unit tests (well, integration
tests
> really) on our persistence stack. We're using HBaseClusterTestCase.
> We have about 25 sets of 20-40 tests that we want to run. Each set needs
to
> stand up a fresh, independent store. We're using maven to do our builds.
>
> 1. In our build machine environment (Ubuntu, Anthill) and my build
> environment (Windows, cygwin), it doesn't work because the test cases
> don't tear down properly, and the test folders stick around, messing up
the
> subsequent tests.
>
> 2. In my IDE (IntelliJ on Windows using IDE's internal maven), the
> tests don't work because MiniDFSCluster eventually calls "du" which
doesn't
> exist.
>
> 3. Anybody have sample code for setting up a simulated data node?
Is
> there such thing as a simulated persistence for zookeeper? (Our test
data
> isn't very large so this would be fine)
>
>
>
> One final question that may at least help my blood pressure: has hbase
unit
> testing improved at all in version 0.90?
>
>
>
> Thanks,
>
> Mark
RE: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
Posted by Jonathan Gray <jg...@fb.com>.
Hey Mark. Sorry to hear about your troubles.
There is a new testing facility that has replaced HBaseClusterTestCase. Check out HBaseTestingUtility. It's JUnit4 based. One sample usage of it is the test TestFromClientSide.
This new one includes support for multiple DataNodes, RegionServers, and Masters. Hopefully that will work better for you.
As far as working in your IDE, du is part of Cygwin. So if it's saying not found, something in your environment (like your path) is not configured properly for Cygwin and is not finding du.
Hope that helps.
JG
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Tuesday, January 18, 2011 2:57 PM
> To: user@hbase.apache.org
> Subject: hbase 0.20.6 - HBaseClusterTestCase, DU, cygwin, IntelliJ - arg!
>
> Sorry about the goofy subject but this is driving me nuts. Any answers to any
> of these questions would be greatly appreciated (well, except a generic
> "yes").
>
>
>
> My goal is to be able to run a sequence of unit tests (well, integration tests
> really) on our persistence stack. We're using HBaseClusterTestCase.
> We have about 25 sets of 20-40 tests that we want to run. Each set needs to
> stand up a fresh, independent store. We're using maven to do our builds.
>
> 1. In our build machine environment (Ubuntu, Anthill) and my build
> environment (Windows, cygwin), it doesn't work because the test cases
> don't tear down properly, and the test folders stick around, messing up the
> subsequent tests.
>
> 2. In my IDE (IntelliJ on Windows using IDE's internal maven), the
> tests don't work because MiniDFSCluster eventually calls "du" which doesn't
> exist.
>
> 3. Anybody have sample code for setting up a simulated data node? Is
> there such thing as a simulated persistence for zookeeper? (Our test data
> isn't very large so this would be fine)
>
>
>
> One final question that may at least help my blood pressure: has hbase unit
> testing improved at all in version 0.90?
>
>
>
> Thanks,
>
> Mark