You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jack Bearden <ja...@jackbearden.com> on 2018/07/31 03:16:33 UTC
Question regarding hbase-shell JRuby testing workflow
Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted to
get some feedback on workflow. My objective was to execute a single Ruby
unit test in isolation from the TestShell.java class via the jruby binary.
I was able to accomplish this by doing the following steps:
1. Pulled down branch-2
2. Installed and cleaned via maven at the base directory (mvn
-Dmaven.javadoc.skip -DskipTests install)
3. Changed to the hbase-shell directory and exported the classpath (mvn
dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
4. Exported the path to that file to shell env (export
TEST_PATH="/path/to/cpath.txt")
5. Hacked tests_runner.rb to just load("path/to/test") for the test I
wanted to run
6. From the hbase-shell project directory ran the following:
jruby \
-J-cp `cat $TEST_PATH` \
-d -w \
-I src/test/ruby \
-I src/main/ruby \
src/test/ruby/tests_runner.rb
The problem is, is this only worked on *most* of the hbase-shell Ruby
tests. The only way to get, for example, list_procedures_test.rb to work
completely, was to run it from the TestShell.java file. When ran from the
jruby binary, I get a "class not found" when
org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was being
referenced. I can't figure out how to load this class adhoc and not through
what appears to be Maven magic.
Any suggestions or better ideas on how to do this?
Re: Question regarding hbase-shell JRuby testing workflow
Posted by Ted Yu <yu...@gmail.com>.
The flakiness of list_procedures_test.rb is probably related to the load on
the node running the test, or other tests in hbase-shell module.
I ran list_procedures_test.rb alone a few times which passed.
Jack:
You can include some other shell test(s) along with this test.
You can also retrieve test output following the test runs performed here:
https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html
FYI
On Tue, Jul 31, 2018 at 7:41 AM Josh Elser <jo...@gmail.com> wrote:
> I haven't ever tried to de-couple from Maven. The 'lowest' I ever got
> was something like the following:
>
> 1. mvn clean install -DskipTests
> 2. cd hbase-shell
> 2. mvn package -Dtest=TestShell -Dshell.test.include=my_test_class.rb -o
>
> Hope this helps, Jack. I know it's not ideal -- if you do come up with
> something that works at a lower level, I think we'd be very supportive
> to get it doc'ed and keep it working :)
>
> On 7/30/18 11:16 PM, Jack Bearden wrote:
> > Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted
> to
> > get some feedback on workflow. My objective was to execute a single Ruby
> > unit test in isolation from the TestShell.java class via the jruby
> binary.
> > I was able to accomplish this by doing the following steps:
> >
> > 1. Pulled down branch-2
> > 2. Installed and cleaned via maven at the base directory (mvn
> > -Dmaven.javadoc.skip -DskipTests install)
> > 3. Changed to the hbase-shell directory and exported the classpath (mvn
> > dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
> > 4. Exported the path to that file to shell env (export
> > TEST_PATH="/path/to/cpath.txt")
> > 5. Hacked tests_runner.rb to just load("path/to/test") for the test I
> > wanted to run
> > 6. From the hbase-shell project directory ran the following:
> >
> > jruby \
> > -J-cp `cat $TEST_PATH` \
> > -d -w \
> > -I src/test/ruby \
> > -I src/main/ruby \
> > src/test/ruby/tests_runner.rb
> >
> > The problem is, is this only worked on *most* of the hbase-shell Ruby
> > tests. The only way to get, for example, list_procedures_test.rb to work
> > completely, was to run it from the TestShell.java file. When ran from the
> > jruby binary, I get a "class not found" when
> > org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was being
> > referenced. I can't figure out how to load this class adhoc and not
> through
> > what appears to be Maven magic.
> >
> > Any suggestions or better ideas on how to do this?
> >
>
Re: Question regarding hbase-shell JRuby testing workflow
Posted by Josh Elser <jo...@gmail.com>.
I haven't ever tried to de-couple from Maven. The 'lowest' I ever got
was something like the following:
1. mvn clean install -DskipTests
2. cd hbase-shell
2. mvn package -Dtest=TestShell -Dshell.test.include=my_test_class.rb -o
Hope this helps, Jack. I know it's not ideal -- if you do come up with
something that works at a lower level, I think we'd be very supportive
to get it doc'ed and keep it working :)
On 7/30/18 11:16 PM, Jack Bearden wrote:
> Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted to
> get some feedback on workflow. My objective was to execute a single Ruby
> unit test in isolation from the TestShell.java class via the jruby binary.
> I was able to accomplish this by doing the following steps:
>
> 1. Pulled down branch-2
> 2. Installed and cleaned via maven at the base directory (mvn
> -Dmaven.javadoc.skip -DskipTests install)
> 3. Changed to the hbase-shell directory and exported the classpath (mvn
> dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
> 4. Exported the path to that file to shell env (export
> TEST_PATH="/path/to/cpath.txt")
> 5. Hacked tests_runner.rb to just load("path/to/test") for the test I
> wanted to run
> 6. From the hbase-shell project directory ran the following:
>
> jruby \
> -J-cp `cat $TEST_PATH` \
> -d -w \
> -I src/test/ruby \
> -I src/main/ruby \
> src/test/ruby/tests_runner.rb
>
> The problem is, is this only worked on *most* of the hbase-shell Ruby
> tests. The only way to get, for example, list_procedures_test.rb to work
> completely, was to run it from the TestShell.java file. When ran from the
> jruby binary, I get a "class not found" when
> org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was being
> referenced. I can't figure out how to load this class adhoc and not through
> what appears to be Maven magic.
>
> Any suggestions or better ideas on how to do this?
>
Re: Question regarding hbase-shell JRuby testing workflow
Posted by Ted Yu <yu...@gmail.com>.
Have you tried sidelining other .rb files
under hbase-shell//src/test/ruby/shell/ (keeping only
hbase-shell//src/test/ruby/shell/list_procedures_test.rb) ?
Cheers
On Mon, Jul 30, 2018 at 8:29 PM Jack Bearden <ja...@jackbearden.com> wrote:
> Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted to
> get some feedback on workflow. My objective was to execute a single Ruby
> unit test in isolation from the TestShell.java class via the jruby binary.
> I was able to accomplish this by doing the following steps:
>
> 1. Pulled down branch-2
> 2. Installed and cleaned via maven at the base directory (mvn
> -Dmaven.javadoc.skip -DskipTests install)
> 3. Changed to the hbase-shell directory and exported the classpath (mvn
> dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
> 4. Exported the path to that file to shell env (export
> TEST_PATH="/path/to/cpath.txt")
> 5. Hacked tests_runner.rb to just load("path/to/test") for the test I
> wanted to run
> 6. From the hbase-shell project directory ran the following:
>
> jruby \
> -J-cp `cat $TEST_PATH` \
> -d -w \
> -I src/test/ruby \
> -I src/main/ruby \
> src/test/ruby/tests_runner.rb
>
> The problem is, is this only worked on *most* of the hbase-shell Ruby
> tests. The only way to get, for example, list_procedures_test.rb to work
> completely, was to run it from the TestShell.java file. When ran from the
> jruby binary, I get a "class not found" when
> org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was being
> referenced. I can't figure out how to load this class adhoc and not through
> what appears to be Maven magic.
>
> Any suggestions or better ideas on how to do this?
>
Re: Question regarding hbase-shell JRuby testing workflow
Posted by Jack Bearden <ja...@altiscale.com>.
Great feedback. Thanks all
On Tue, Jul 31, 2018 at 10:59 AM, Sean Busbey <bu...@apache.org> wrote:
> check out the configs for the maven-dependency-plugin. My guess would
> be that your call to dependency:build-classpath is getting the
> artifacts needed for runtime scope and not test scope.
>
> On Mon, Jul 30, 2018 at 10:16 PM, Jack Bearden <ja...@jackbearden.com>
> wrote:
> > Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted
> to
> > get some feedback on workflow. My objective was to execute a single Ruby
> > unit test in isolation from the TestShell.java class via the jruby
> binary.
> > I was able to accomplish this by doing the following steps:
> >
> > 1. Pulled down branch-2
> > 2. Installed and cleaned via maven at the base directory (mvn
> > -Dmaven.javadoc.skip -DskipTests install)
> > 3. Changed to the hbase-shell directory and exported the classpath (mvn
> > dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
> > 4. Exported the path to that file to shell env (export
> > TEST_PATH="/path/to/cpath.txt")
> > 5. Hacked tests_runner.rb to just load("path/to/test") for the test I
> > wanted to run
> > 6. From the hbase-shell project directory ran the following:
> >
> > jruby \
> > -J-cp `cat $TEST_PATH` \
> > -d -w \
> > -I src/test/ruby \
> > -I src/main/ruby \
> > src/test/ruby/tests_runner.rb
> >
> > The problem is, is this only worked on *most* of the hbase-shell Ruby
> > tests. The only way to get, for example, list_procedures_test.rb to work
> > completely, was to run it from the TestShell.java file. When ran from the
> > jruby binary, I get a "class not found" when
> > org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was
> being
> > referenced. I can't figure out how to load this class adhoc and not
> through
> > what appears to be Maven magic.
> >
> > Any suggestions or better ideas on how to do this?
>
Re: Question regarding hbase-shell JRuby testing workflow
Posted by Sean Busbey <bu...@apache.org>.
check out the configs for the maven-dependency-plugin. My guess would
be that your call to dependency:build-classpath is getting the
artifacts needed for runtime scope and not test scope.
On Mon, Jul 30, 2018 at 10:16 PM, Jack Bearden <ja...@jackbearden.com> wrote:
> Hey all! I was hacking hbase-shell and JRuby over the weekend and wanted to
> get some feedback on workflow. My objective was to execute a single Ruby
> unit test in isolation from the TestShell.java class via the jruby binary.
> I was able to accomplish this by doing the following steps:
>
> 1. Pulled down branch-2
> 2. Installed and cleaned via maven at the base directory (mvn
> -Dmaven.javadoc.skip -DskipTests install)
> 3. Changed to the hbase-shell directory and exported the classpath (mvn
> dependency:build-classpath -Dmdep.outputFile=/path/to/cpath.txt)
> 4. Exported the path to that file to shell env (export
> TEST_PATH="/path/to/cpath.txt")
> 5. Hacked tests_runner.rb to just load("path/to/test") for the test I
> wanted to run
> 6. From the hbase-shell project directory ran the following:
>
> jruby \
> -J-cp `cat $TEST_PATH` \
> -d -w \
> -I src/test/ruby \
> -I src/main/ruby \
> src/test/ruby/tests_runner.rb
>
> The problem is, is this only worked on *most* of the hbase-shell Ruby
> tests. The only way to get, for example, list_procedures_test.rb to work
> completely, was to run it from the TestShell.java file. When ran from the
> jruby binary, I get a "class not found" when
> org.apache.hadoop.hbase.client.procedure.ShellTestProcedure.new was being
> referenced. I can't figure out how to load this class adhoc and not through
> what appears to be Maven magic.
>
> Any suggestions or better ideas on how to do this?