You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by Tim Robertson <ti...@gmail.com> on 2018/08/02 15:42:49 UTC
Re: Binaries for embedded testing
Hi folks
I've not had too much time, but I threw this together using Mike's
binaries:
https://github.com/timrobertson100/kudu-test-server
I think this shows that running a mini cluster is possible using the
binaries Mike prepared when they are included in a jar (on CentOS 7.4 at
least).
Please don't flame me for the code - it was a rush job - but perhaps you
could verify it works for you Mike?
On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org> wrote:
> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <ti...@gmail.com>
> wrote:
>
> > > I'm pretty busy this week and would be happy to get some help on this
> if
> > anybody has cycles to spare.
> >
> > Thanks Mike - I'll look into 3) and if we get it working I'm happy to
> offer
> > a PR for 4).
> > This is an evening project for me so I might be a little slow too - if
> > someone else is keen and has time please feel free to jump in.
> >
>
> Sounds great! I think Grant started working on #4 but I don't know how far
> he got.
>
> Mike
>
Re: Binaries for embedded testing
Posted by Mike Percy <mp...@apache.org>.
I wanted to post a follow-up here with an update from me on current
progress and propose some next steps for publishing / using relocatable
artifacts in the mini cluster so we can collaborate on completing this.
1. The patch to build the artifacts is in review and I need to finish
incorporating review feedback and publish a new revision at
https://gerrit.cloudera.org/c/11377/
2. I wrote a proof-of-concept to find something like the above artifact in
the Java classpath and unpack it to the current directory, which I posted
to GitHub at https://github.com/mpercy/resource-jar-tools
Some of the issues I noticed when implementing the POC in #2 are:
1. How to identify the specific artifact we are looking for without the
potential for some accidental conflict with another JAR
2. How to choose between them if we find multiple artifacts on the
classpath or we end up with multiple operating system artifacts / the wrong
os artifact on the classpath
3. How to make this extensible in case we want to be able to handle
multiple versions on the classpath in the future, i.e. for upgrade / compat
testing purposes
Based on the above experiment in #2 I think we should make a couple changes
to the format produced by #1 in order to make the interface between #2 and
#1 more extensible and explicit.
So I was thinking that the format of the archive can be very similar to a
typical binary release tarball layout plus a top-level META-INF directory
with a properties file in the META-INF directory with a specific file name.
The properties file will be what we look for when searching the classpath.
JAR file layout:
META-INF/apache-kudu-test-binary.properties
apache-kudu-test-binary-1.8.0/bin/kudu-tserver
apache-kudu-test-binary-1.8.0/lib/libkrpc.so
apache-kudu-test-binary-1.8.0/lib/libz.so.1
apache-kudu-test-binary-1.8.0/lib/libcrypto.so.1.1
...
The META-INF/apache-kudu-test-binary.properties file would contain
everything the unpacking code needs to know about where to find the files
and would be extensible in case we want to support fancier features in the
future.
Example properties file contents:
$ cat META-INF/apache-kudu-test-binary.properties
format.version=1
artifact.version=1.8.0
artifact.prefix=apache-kudu-test-binary-1.8.0
artifact.os=linux
artifact.arch=x86_64
So the semantics of the above would be:
- format.version is the format of this artifact, in case we need to extend
it later and be able to tell the difference between them
- artifact.version is the version of the release (or snapshot, i.e.
1.9.0-SNAPSHOT) in case we want to support multiple versions later or want
the client to be able to request a specific version
- artifact.prefix is the directory name at the root of the JAR under which
the relevant files can be found (think ./configure --prefix equivalent when
building autoconf software) -- we unpack from here
- artifact.os is the operating system and we will use the same convention
as https://github.com/trustin/os-maven-plugin for ${os.detected.name} which
practically speaking for Kudu, just means { linux | osx }.
- artifact.arch is the target system architecture and again we will use
the same convention as the above os-maven-plugin for ${os.detected.arch}
which, practically speaking for Kudu, means this will always be set
to x86_64.
Regarding the software to find the JAR on the classpath and unpack it,
based on the above format we could use the prototype code at
https://github.com/mpercy/resource-jar-tools to unpack it by running
something like this:
$ java -cp
target/ResourceJarTools-1.0-SNAPSHOT.jar:$HOME/apache-kudu-test-binary-1.7.0.jar
org.apache.kudu.ListJarFromResource
META-INF/apache-kudu-test-binary.properties
$ java -cp
target/ResourceJarTools-1.0-SNAPSHOT.jar:$HOME/apache-kudu-test-binary-1.7.0.jar
org.apache.kudu.UnpackJarFromResource
META-INF/apache-kudu-test-binary.properties .
... assuming we were in the root dir of that project and had a test binary
jar in $HOME. But that code will simply unpack the whole jar into the
current directory and I think what we want is to unpack everything under
${artifact.prefix} to whatever temp dir we create for starting the
minicluster.
In terms of the API, Grant and I were musing on this in #community-discuss
a week or two ago and we sort of landed on the following simple API (I took
a little editorial license with the method names here):
/** Returns true if there is a test binary jar on the classpath */
public static bool isKuduTestBinaryJarOnClasspath();
/** Unpack the first located Kudu test binary JAR found on the classpath
into destDir. Throws if none found or unpacking fails. */
public static void unpackFirstTestKuduBinary(String destDir);
In the future, this could be extended to list available versions, unpack
specific versions, etc, but that seems unnecessary for an initial
implementation.
Finally, in terms of how this works at a high level, so far we thought that
this would unpack the files to a temporary directory and automatically
delete all of the the unpacked files and directories on JVM exit. We would
also have to put in a bit of plumbing to decide when to use this
classpath-searching approach (good for projects to integrate with) vs.
using a location determined in advance (like the current Kudu unit tests
do, and should continue to do).
So how does the test binary JAR artifact end up on the classpath in the
first place? Any project that needs that would just add it as a test
dependency to their build system and Maven / Gradle should automatically
take care of that part, so nothing special there aside from some additional
work at release time.
Please let me know if I left anything out. Let's discuss further here in
this thread or on Slack.
Mike
On Wed, Oct 24, 2018 at 2:15 PM Grant Henke <gh...@cloudera.com.invalid>
wrote:
> I wanted to provide a detailed update on the public testing utilities work
> that we have been doing since this email thread started to hopefully
> continue the discussion and motivate wrapping things up.
>
> Recently I restructured the binary location logic to make it a bit more
> friendly to external projects. We also adjusted our testing base class to
> be a rule so it could be composed into other tests. Then shortly after
> branching Kudu 1.8 we broke out the testing utilities into their own jar.
> The idea, is that Kudu 1.9 would contain some public testing artifacts that
> other projects could use.
>
> Some of those relevant commits are:
>
> - [test] Clean up MiniKuduCluster and BaseKuduTest
> <
> https://github.com/apache/kudu/commit/fd1ffd0fb65e138f1f015a55aa96ae870c1d51cd
> >
> - [test] Adjust Kudu binary locator logic.
> <
> https://github.com/apache/kudu/commit/34e88d3dafc421ccaabae5767aad2fd9fa015d39
> >
> - [test] Move BaseKuduTest to a Junit Rule
> <
> https://github.com/apache/kudu/commit/dc8ae79961f71b8bdc344781fc89d38d94152fc4
> >
> - KUDU-2411: (Part 1) Break out existing test utilities into a seperate
> module
> <
> https://github.com/apache/kudu/commit/15f1416f67dcb714842d02647a1f2e06e675660d
> >
> - [java] Allow command line override of kuduBinDir
> <
> https://github.com/apache/kudu/commit/785490ce509e68029f8062882bb7021895e3b446
> >
>
> I tested Mike's patch (here <https://gerrit.cloudera.org/#/c/11377/>) and
> the generated artifact using -DkuduBinDir and things look like they will
> work. It seams like we need to find a way to package this stuff up, deploy
> it, and then write a utility class to locate it on the classpath and call
> it.
>
> Recently we have been looking at the way protoc is built and published
> <https://github.com/protocolbuffers/protobuf/tree/master/protoc-artifact>
> for inspiration and to see how they handle OSX. Tim's project
> <https://github.com/timrobertson100/kudu-test-server> is a good example of
> finding it on the classpath and executing it.
>
> Combining those 2 things we should have something useable.
>
> Any feedback, tips, suggestion or collaboration is more than welcome.
>
> Thanks,
> Grant
>
> On Sat, Aug 18, 2018 at 5:40 AM Tim Robertson <ti...@gmail.com>
> wrote:
>
> > I've updated the GH to reflect the idea outlined above Mike
> > https://github.com/timrobertson100/kudu-test-server
> >
> > - code is fairly hacky
> > - deletes extracted binaries (FYI: deleteOnExit() will not delete dirs
> > unless empty)
> > - uses maven classifier (only linux exists)
> >
> > Can you PTAL Mike when you get a chance? - no rush at all
> >
> >
> >
> > On Thu, Aug 9, 2018 at 6:07 PM Tim <ti...@gmail.com> wrote:
> >
> > > Thanks Mike - no apologies needed at all
> > >
> > > I’ll aim to reshape the GH repo I did to illustrate what we outlined,
> > with
> > > an example of use.
> > >
> > > Outstanding is some OS X binaries if anyone has some time?
> > >
> > > Have a good trip.
> > >
> > > Tim,
> > > Sent from my iPhone
> > >
> > > > On 9 Aug 2018, at 14:05, Mike Percy <mp...@apache.org> wrote:
> > > >
> > > > Hi Tim,
> > > > Sorry for the delay in responding, I've been trying to get back to
> > this.
> > > > You make some great points. If we can forego Maven/Gradle plugins, we
> > can
> > > > do this with a lot less work. Initially, I was concerned about
> > unpacking
> > > > the bits potentially more than strictly necessary, but it seems
> likely
> > > that
> > > > the CPU / IO required to do the unpacking would probably not be
> > > noticeable
> > > > in the grand scheme of running tests against a Kudu MiniCluster,
> > > especially
> > > > if it's only done once per test file (using @BeforeClass or similar).
> > > Also,
> > > > as long as there is some way to clean up the files that were unpacked
> > and
> > > > avoid potentially filling up a temp dir then nobody should have a
> > problem
> > > > with this approach... perhaps we can register a JVM shutdown hook and
> > > also
> > > > provide some kind of teardown() method so people can ensure the files
> > get
> > > > cleaned up as appropriate (I saw the TODO in your test code for this;
> > > > protoc uses File.deleteOnExit()).
> > > >
> > > > Regarding creating the binary artifacts, I don't think it's too big
> of
> > a
> > > > burden to ask the release manager to upload either a RHEL 6 or macOS
> > > binary
> > > > test artifact based on the output of a script, and ask the PMC for
> help
> > > to
> > > > get a binary for the other platform.
> > > >
> > > > I'm out of town for the next couple of weeks but once I get back I'll
> > see
> > > > if I can push this forward a bit more.
> > > >
> > > > Thanks!
> > > > Mike
> > > >
> > > >
> > > > On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <
> > timrobertson100@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Thanks to you for making this test possible Mike.
> > > >>
> > > >> I was approaching this emulating protoc where they put the binaries
> as
> > > >> artifacts for Maven [1].
> > > >> A slight difference is that protoc is a single file while your
> tarball
> > > has
> > > >> several. I notice that the protoc build plugin for maven also copies
> > out
> > > >> from the jar to a local filesystem [2] so I just copied that
> approach.
> > > >>
> > > >> What I could imagine is we have a jar with the binaries and a single
> > > class
> > > >> (say EmbeddedKudu) that copies the binaries into a temporary
> directory
> > > (as
> > > >> my hack in GH).
> > > >>
> > > >> A user would then do the following:
> > > >>
> > > >> <!-- finds the environment -->
> > > >> <build>
> > > >> <extensions>
> > > >> <extension>
> > > >> <groupId>kr.motd.maven</groupId>
> > > >> <artifactId>os-maven-plugin</artifactId>
> > > >> <version>1.6.0</version>
> > > >> </extension>
> > > >> </extensions>
> > > >> </build>
> > > >> ....
> > > >> <!-- Adds mini cluster -->
> > > >> <dependency>
> > > >> <groupId>org.apache.kudu</groupId>
> > > >> <artifactId>kudu-client</artifactId>
> > > >> <version>1.7.0</version>
> > > >> <classifier>tests</classifier>
> > > >> </dependency>
> > > >> <!-- Adds an embedded Kudu -->
> > > >> <dependency>
> > > >> <groupId>org.apache.kudu</groupId>
> > > >> <artifactId>kudu-test-server</artifactId>
> > > >> <version>1.7.0</version>
> > > >> <classifier>${os.detected.classifier}</classifier>
> > > >> </dependency>
> > > >>
> > > >>
> > > >> The detect classifier stuff is copied from protoc, but perhaps we'd
> > just
> > > >> state that available options are (?) linux / OSX and ignore
> > > autodetection.
> > > >>
> > > >> And in code users would do something like:
> > > >>
> > > >> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
> > > >> system variable (as per my example)
> > > >> MiniKuduCluster miniCluster =
> > > >> new
> > > >>
> > >
> >
> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
> > > >>
> > > >>
> > > >> What this would mean:
> > > >>
> > > >> - no change to the existing Kudu code packkaging
> > > >> - leverage caching of the binaries as any Java artifact
> > > >> - download would be at build time not test runtime - only copying
> out
> > > from
> > > >> the jar each time
> > > >> - simple to include in most build environments (not tied to maven
> as a
> > > >> plugin)
> > > >>
> > > >> I can't comment if it makes sense to do this WRT the binaries though
> > > and it
> > > >> would mean building and releasing binaries into a jar on each Kudu
> > > release
> > > >> (as protoc does).
> > > >>
> > > >> WDYT?
> > > >>
> > > >> Thanks,
> > > >> Tim
> > > >>
> > > >> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
> > > >> [2]
> > > >>
> > > >>
> > >
> >
> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
> > > >>
> > > >>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org>
> > wrote:
> > > >>>
> > > >>> Ha, neat, thanks for posting this, Tim. It's a nice proof of
> concept.
> > > >>>
> > > >>> I was imagining that we would try to implement the downloading part
> > as
> > > a
> > > >>> Maven plugin, but maybe it could work to try to download the
> > artifacts
> > > at
> > > >>> runtime with a JUnit test. Do you think we could cache the
> artifacts
> > > >>> somewhere, maybe in the Maven repo somehow, so we don't have to
> > > download
> > > >>> the artifact every time we want to use it? I was hoping to simply
> be
> > > able
> > > >>> to ship a tarball or a jar of the binaries separately from the test
> > > >>> framework code.
> > > >>>
> > > >>> Mike
> > > >>>
> > > >>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <
> > > timrobertson100@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi folks
> > > >>>>
> > > >>>> I've not had too much time, but I threw this together using Mike's
> > > >>>> binaries:
> > > >>>> https://github.com/timrobertson100/kudu-test-server
> > > >>>>
> > > >>>> I think this shows that running a mini cluster is possible using
> the
> > > >>>> binaries Mike prepared when they are included in a jar (on CentOS
> > 7.4
> > > >> at
> > > >>>> least).
> > > >>>>
> > > >>>> Please don't flame me for the code - it was a rush job - but
> perhaps
> > > >> you
> > > >>>> could verify it works for you Mike?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org>
> > > >> wrote:
> > > >>>>
> > > >>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
> > > >>> timrobertson100@gmail.com
> > > >>>>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>>> I'm pretty busy this week and would be happy to get some help
> on
> > > >>> this
> > > >>>>> if
> > > >>>>>> anybody has cycles to spare.
> > > >>>>>>
> > > >>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm
> happy
> > > >> to
> > > >>>>> offer
> > > >>>>>> a PR for 4).
> > > >>>>>> This is an evening project for me so I might be a little slow
> too
> > -
> > > >>> if
> > > >>>>>> someone else is keen and has time please feel free to jump in.
> > > >>>>>>
> > > >>>>>
> > > >>>>> Sounds great! I think Grant started working on #4 but I don't
> know
> > > >> how
> > > >>>> far
> > > >>>>> he got.
> > > >>>>>
> > > >>>>> Mike
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
Re: Binaries for embedded testing
Posted by Grant Henke <gh...@cloudera.com.INVALID>.
I wanted to provide a detailed update on the public testing utilities work
that we have been doing since this email thread started to hopefully
continue the discussion and motivate wrapping things up.
Recently I restructured the binary location logic to make it a bit more
friendly to external projects. We also adjusted our testing base class to
be a rule so it could be composed into other tests. Then shortly after
branching Kudu 1.8 we broke out the testing utilities into their own jar.
The idea, is that Kudu 1.9 would contain some public testing artifacts that
other projects could use.
Some of those relevant commits are:
- [test] Clean up MiniKuduCluster and BaseKuduTest
<https://github.com/apache/kudu/commit/fd1ffd0fb65e138f1f015a55aa96ae870c1d51cd>
- [test] Adjust Kudu binary locator logic.
<https://github.com/apache/kudu/commit/34e88d3dafc421ccaabae5767aad2fd9fa015d39>
- [test] Move BaseKuduTest to a Junit Rule
<https://github.com/apache/kudu/commit/dc8ae79961f71b8bdc344781fc89d38d94152fc4>
- KUDU-2411: (Part 1) Break out existing test utilities into a seperate
module
<https://github.com/apache/kudu/commit/15f1416f67dcb714842d02647a1f2e06e675660d>
- [java] Allow command line override of kuduBinDir
<https://github.com/apache/kudu/commit/785490ce509e68029f8062882bb7021895e3b446>
I tested Mike's patch (here <https://gerrit.cloudera.org/#/c/11377/>) and
the generated artifact using -DkuduBinDir and things look like they will
work. It seams like we need to find a way to package this stuff up, deploy
it, and then write a utility class to locate it on the classpath and call
it.
Recently we have been looking at the way protoc is built and published
<https://github.com/protocolbuffers/protobuf/tree/master/protoc-artifact>
for inspiration and to see how they handle OSX. Tim's project
<https://github.com/timrobertson100/kudu-test-server> is a good example of
finding it on the classpath and executing it.
Combining those 2 things we should have something useable.
Any feedback, tips, suggestion or collaboration is more than welcome.
Thanks,
Grant
On Sat, Aug 18, 2018 at 5:40 AM Tim Robertson <ti...@gmail.com>
wrote:
> I've updated the GH to reflect the idea outlined above Mike
> https://github.com/timrobertson100/kudu-test-server
>
> - code is fairly hacky
> - deletes extracted binaries (FYI: deleteOnExit() will not delete dirs
> unless empty)
> - uses maven classifier (only linux exists)
>
> Can you PTAL Mike when you get a chance? - no rush at all
>
>
>
> On Thu, Aug 9, 2018 at 6:07 PM Tim <ti...@gmail.com> wrote:
>
> > Thanks Mike - no apologies needed at all
> >
> > I’ll aim to reshape the GH repo I did to illustrate what we outlined,
> with
> > an example of use.
> >
> > Outstanding is some OS X binaries if anyone has some time?
> >
> > Have a good trip.
> >
> > Tim,
> > Sent from my iPhone
> >
> > > On 9 Aug 2018, at 14:05, Mike Percy <mp...@apache.org> wrote:
> > >
> > > Hi Tim,
> > > Sorry for the delay in responding, I've been trying to get back to
> this.
> > > You make some great points. If we can forego Maven/Gradle plugins, we
> can
> > > do this with a lot less work. Initially, I was concerned about
> unpacking
> > > the bits potentially more than strictly necessary, but it seems likely
> > that
> > > the CPU / IO required to do the unpacking would probably not be
> > noticeable
> > > in the grand scheme of running tests against a Kudu MiniCluster,
> > especially
> > > if it's only done once per test file (using @BeforeClass or similar).
> > Also,
> > > as long as there is some way to clean up the files that were unpacked
> and
> > > avoid potentially filling up a temp dir then nobody should have a
> problem
> > > with this approach... perhaps we can register a JVM shutdown hook and
> > also
> > > provide some kind of teardown() method so people can ensure the files
> get
> > > cleaned up as appropriate (I saw the TODO in your test code for this;
> > > protoc uses File.deleteOnExit()).
> > >
> > > Regarding creating the binary artifacts, I don't think it's too big of
> a
> > > burden to ask the release manager to upload either a RHEL 6 or macOS
> > binary
> > > test artifact based on the output of a script, and ask the PMC for help
> > to
> > > get a binary for the other platform.
> > >
> > > I'm out of town for the next couple of weeks but once I get back I'll
> see
> > > if I can push this forward a bit more.
> > >
> > > Thanks!
> > > Mike
> > >
> > >
> > > On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <
> timrobertson100@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks to you for making this test possible Mike.
> > >>
> > >> I was approaching this emulating protoc where they put the binaries as
> > >> artifacts for Maven [1].
> > >> A slight difference is that protoc is a single file while your tarball
> > has
> > >> several. I notice that the protoc build plugin for maven also copies
> out
> > >> from the jar to a local filesystem [2] so I just copied that approach.
> > >>
> > >> What I could imagine is we have a jar with the binaries and a single
> > class
> > >> (say EmbeddedKudu) that copies the binaries into a temporary directory
> > (as
> > >> my hack in GH).
> > >>
> > >> A user would then do the following:
> > >>
> > >> <!-- finds the environment -->
> > >> <build>
> > >> <extensions>
> > >> <extension>
> > >> <groupId>kr.motd.maven</groupId>
> > >> <artifactId>os-maven-plugin</artifactId>
> > >> <version>1.6.0</version>
> > >> </extension>
> > >> </extensions>
> > >> </build>
> > >> ....
> > >> <!-- Adds mini cluster -->
> > >> <dependency>
> > >> <groupId>org.apache.kudu</groupId>
> > >> <artifactId>kudu-client</artifactId>
> > >> <version>1.7.0</version>
> > >> <classifier>tests</classifier>
> > >> </dependency>
> > >> <!-- Adds an embedded Kudu -->
> > >> <dependency>
> > >> <groupId>org.apache.kudu</groupId>
> > >> <artifactId>kudu-test-server</artifactId>
> > >> <version>1.7.0</version>
> > >> <classifier>${os.detected.classifier}</classifier>
> > >> </dependency>
> > >>
> > >>
> > >> The detect classifier stuff is copied from protoc, but perhaps we'd
> just
> > >> state that available options are (?) linux / OSX and ignore
> > autodetection.
> > >>
> > >> And in code users would do something like:
> > >>
> > >> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
> > >> system variable (as per my example)
> > >> MiniKuduCluster miniCluster =
> > >> new
> > >>
> >
> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
> > >>
> > >>
> > >> What this would mean:
> > >>
> > >> - no change to the existing Kudu code packkaging
> > >> - leverage caching of the binaries as any Java artifact
> > >> - download would be at build time not test runtime - only copying out
> > from
> > >> the jar each time
> > >> - simple to include in most build environments (not tied to maven as a
> > >> plugin)
> > >>
> > >> I can't comment if it makes sense to do this WRT the binaries though
> > and it
> > >> would mean building and releasing binaries into a jar on each Kudu
> > release
> > >> (as protoc does).
> > >>
> > >> WDYT?
> > >>
> > >> Thanks,
> > >> Tim
> > >>
> > >> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
> > >> [2]
> > >>
> > >>
> >
> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
> > >>
> > >>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org>
> wrote:
> > >>>
> > >>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
> > >>>
> > >>> I was imagining that we would try to implement the downloading part
> as
> > a
> > >>> Maven plugin, but maybe it could work to try to download the
> artifacts
> > at
> > >>> runtime with a JUnit test. Do you think we could cache the artifacts
> > >>> somewhere, maybe in the Maven repo somehow, so we don't have to
> > download
> > >>> the artifact every time we want to use it? I was hoping to simply be
> > able
> > >>> to ship a tarball or a jar of the binaries separately from the test
> > >>> framework code.
> > >>>
> > >>> Mike
> > >>>
> > >>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <
> > timrobertson100@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi folks
> > >>>>
> > >>>> I've not had too much time, but I threw this together using Mike's
> > >>>> binaries:
> > >>>> https://github.com/timrobertson100/kudu-test-server
> > >>>>
> > >>>> I think this shows that running a mini cluster is possible using the
> > >>>> binaries Mike prepared when they are included in a jar (on CentOS
> 7.4
> > >> at
> > >>>> least).
> > >>>>
> > >>>> Please don't flame me for the code - it was a rush job - but perhaps
> > >> you
> > >>>> could verify it works for you Mike?
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org>
> > >> wrote:
> > >>>>
> > >>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
> > >>> timrobertson100@gmail.com
> > >>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>>> I'm pretty busy this week and would be happy to get some help on
> > >>> this
> > >>>>> if
> > >>>>>> anybody has cycles to spare.
> > >>>>>>
> > >>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy
> > >> to
> > >>>>> offer
> > >>>>>> a PR for 4).
> > >>>>>> This is an evening project for me so I might be a little slow too
> -
> > >>> if
> > >>>>>> someone else is keen and has time please feel free to jump in.
> > >>>>>>
> > >>>>>
> > >>>>> Sounds great! I think Grant started working on #4 but I don't know
> > >> how
> > >>>> far
> > >>>>> he got.
> > >>>>>
> > >>>>> Mike
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: Binaries for embedded testing
Posted by Tim Robertson <ti...@gmail.com>.
I've updated the GH to reflect the idea outlined above Mike
https://github.com/timrobertson100/kudu-test-server
- code is fairly hacky
- deletes extracted binaries (FYI: deleteOnExit() will not delete dirs
unless empty)
- uses maven classifier (only linux exists)
Can you PTAL Mike when you get a chance? - no rush at all
On Thu, Aug 9, 2018 at 6:07 PM Tim <ti...@gmail.com> wrote:
> Thanks Mike - no apologies needed at all
>
> I’ll aim to reshape the GH repo I did to illustrate what we outlined, with
> an example of use.
>
> Outstanding is some OS X binaries if anyone has some time?
>
> Have a good trip.
>
> Tim,
> Sent from my iPhone
>
> > On 9 Aug 2018, at 14:05, Mike Percy <mp...@apache.org> wrote:
> >
> > Hi Tim,
> > Sorry for the delay in responding, I've been trying to get back to this.
> > You make some great points. If we can forego Maven/Gradle plugins, we can
> > do this with a lot less work. Initially, I was concerned about unpacking
> > the bits potentially more than strictly necessary, but it seems likely
> that
> > the CPU / IO required to do the unpacking would probably not be
> noticeable
> > in the grand scheme of running tests against a Kudu MiniCluster,
> especially
> > if it's only done once per test file (using @BeforeClass or similar).
> Also,
> > as long as there is some way to clean up the files that were unpacked and
> > avoid potentially filling up a temp dir then nobody should have a problem
> > with this approach... perhaps we can register a JVM shutdown hook and
> also
> > provide some kind of teardown() method so people can ensure the files get
> > cleaned up as appropriate (I saw the TODO in your test code for this;
> > protoc uses File.deleteOnExit()).
> >
> > Regarding creating the binary artifacts, I don't think it's too big of a
> > burden to ask the release manager to upload either a RHEL 6 or macOS
> binary
> > test artifact based on the output of a script, and ask the PMC for help
> to
> > get a binary for the other platform.
> >
> > I'm out of town for the next couple of weeks but once I get back I'll see
> > if I can push this forward a bit more.
> >
> > Thanks!
> > Mike
> >
> >
> > On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <timrobertson100@gmail.com
> >
> > wrote:
> >
> >> Thanks to you for making this test possible Mike.
> >>
> >> I was approaching this emulating protoc where they put the binaries as
> >> artifacts for Maven [1].
> >> A slight difference is that protoc is a single file while your tarball
> has
> >> several. I notice that the protoc build plugin for maven also copies out
> >> from the jar to a local filesystem [2] so I just copied that approach.
> >>
> >> What I could imagine is we have a jar with the binaries and a single
> class
> >> (say EmbeddedKudu) that copies the binaries into a temporary directory
> (as
> >> my hack in GH).
> >>
> >> A user would then do the following:
> >>
> >> <!-- finds the environment -->
> >> <build>
> >> <extensions>
> >> <extension>
> >> <groupId>kr.motd.maven</groupId>
> >> <artifactId>os-maven-plugin</artifactId>
> >> <version>1.6.0</version>
> >> </extension>
> >> </extensions>
> >> </build>
> >> ....
> >> <!-- Adds mini cluster -->
> >> <dependency>
> >> <groupId>org.apache.kudu</groupId>
> >> <artifactId>kudu-client</artifactId>
> >> <version>1.7.0</version>
> >> <classifier>tests</classifier>
> >> </dependency>
> >> <!-- Adds an embedded Kudu -->
> >> <dependency>
> >> <groupId>org.apache.kudu</groupId>
> >> <artifactId>kudu-test-server</artifactId>
> >> <version>1.7.0</version>
> >> <classifier>${os.detected.classifier}</classifier>
> >> </dependency>
> >>
> >>
> >> The detect classifier stuff is copied from protoc, but perhaps we'd just
> >> state that available options are (?) linux / OSX and ignore
> autodetection.
> >>
> >> And in code users would do something like:
> >>
> >> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
> >> system variable (as per my example)
> >> MiniKuduCluster miniCluster =
> >> new
> >>
> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
> >>
> >>
> >> What this would mean:
> >>
> >> - no change to the existing Kudu code packkaging
> >> - leverage caching of the binaries as any Java artifact
> >> - download would be at build time not test runtime - only copying out
> from
> >> the jar each time
> >> - simple to include in most build environments (not tied to maven as a
> >> plugin)
> >>
> >> I can't comment if it makes sense to do this WRT the binaries though
> and it
> >> would mean building and releasing binaries into a jar on each Kudu
> release
> >> (as protoc does).
> >>
> >> WDYT?
> >>
> >> Thanks,
> >> Tim
> >>
> >> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
> >> [2]
> >>
> >>
> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
> >>
> >>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org> wrote:
> >>>
> >>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
> >>>
> >>> I was imagining that we would try to implement the downloading part as
> a
> >>> Maven plugin, but maybe it could work to try to download the artifacts
> at
> >>> runtime with a JUnit test. Do you think we could cache the artifacts
> >>> somewhere, maybe in the Maven repo somehow, so we don't have to
> download
> >>> the artifact every time we want to use it? I was hoping to simply be
> able
> >>> to ship a tarball or a jar of the binaries separately from the test
> >>> framework code.
> >>>
> >>> Mike
> >>>
> >>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <
> timrobertson100@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi folks
> >>>>
> >>>> I've not had too much time, but I threw this together using Mike's
> >>>> binaries:
> >>>> https://github.com/timrobertson100/kudu-test-server
> >>>>
> >>>> I think this shows that running a mini cluster is possible using the
> >>>> binaries Mike prepared when they are included in a jar (on CentOS 7.4
> >> at
> >>>> least).
> >>>>
> >>>> Please don't flame me for the code - it was a rush job - but perhaps
> >> you
> >>>> could verify it works for you Mike?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org>
> >> wrote:
> >>>>
> >>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
> >>> timrobertson100@gmail.com
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>>> I'm pretty busy this week and would be happy to get some help on
> >>> this
> >>>>> if
> >>>>>> anybody has cycles to spare.
> >>>>>>
> >>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy
> >> to
> >>>>> offer
> >>>>>> a PR for 4).
> >>>>>> This is an evening project for me so I might be a little slow too -
> >>> if
> >>>>>> someone else is keen and has time please feel free to jump in.
> >>>>>>
> >>>>>
> >>>>> Sounds great! I think Grant started working on #4 but I don't know
> >> how
> >>>> far
> >>>>> he got.
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>
> >>>
> >>
>
Re: Binaries for embedded testing
Posted by Tim <ti...@gmail.com>.
Thanks Mike - no apologies needed at all
I’ll aim to reshape the GH repo I did to illustrate what we outlined, with an example of use.
Outstanding is some OS X binaries if anyone has some time?
Have a good trip.
Tim,
Sent from my iPhone
> On 9 Aug 2018, at 14:05, Mike Percy <mp...@apache.org> wrote:
>
> Hi Tim,
> Sorry for the delay in responding, I've been trying to get back to this.
> You make some great points. If we can forego Maven/Gradle plugins, we can
> do this with a lot less work. Initially, I was concerned about unpacking
> the bits potentially more than strictly necessary, but it seems likely that
> the CPU / IO required to do the unpacking would probably not be noticeable
> in the grand scheme of running tests against a Kudu MiniCluster, especially
> if it's only done once per test file (using @BeforeClass or similar). Also,
> as long as there is some way to clean up the files that were unpacked and
> avoid potentially filling up a temp dir then nobody should have a problem
> with this approach... perhaps we can register a JVM shutdown hook and also
> provide some kind of teardown() method so people can ensure the files get
> cleaned up as appropriate (I saw the TODO in your test code for this;
> protoc uses File.deleteOnExit()).
>
> Regarding creating the binary artifacts, I don't think it's too big of a
> burden to ask the release manager to upload either a RHEL 6 or macOS binary
> test artifact based on the output of a script, and ask the PMC for help to
> get a binary for the other platform.
>
> I'm out of town for the next couple of weeks but once I get back I'll see
> if I can push this forward a bit more.
>
> Thanks!
> Mike
>
>
> On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <ti...@gmail.com>
> wrote:
>
>> Thanks to you for making this test possible Mike.
>>
>> I was approaching this emulating protoc where they put the binaries as
>> artifacts for Maven [1].
>> A slight difference is that protoc is a single file while your tarball has
>> several. I notice that the protoc build plugin for maven also copies out
>> from the jar to a local filesystem [2] so I just copied that approach.
>>
>> What I could imagine is we have a jar with the binaries and a single class
>> (say EmbeddedKudu) that copies the binaries into a temporary directory (as
>> my hack in GH).
>>
>> A user would then do the following:
>>
>> <!-- finds the environment -->
>> <build>
>> <extensions>
>> <extension>
>> <groupId>kr.motd.maven</groupId>
>> <artifactId>os-maven-plugin</artifactId>
>> <version>1.6.0</version>
>> </extension>
>> </extensions>
>> </build>
>> ....
>> <!-- Adds mini cluster -->
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-client</artifactId>
>> <version>1.7.0</version>
>> <classifier>tests</classifier>
>> </dependency>
>> <!-- Adds an embedded Kudu -->
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-test-server</artifactId>
>> <version>1.7.0</version>
>> <classifier>${os.detected.classifier}</classifier>
>> </dependency>
>>
>>
>> The detect classifier stuff is copied from protoc, but perhaps we'd just
>> state that available options are (?) linux / OSX and ignore autodetection.
>>
>> And in code users would do something like:
>>
>> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
>> system variable (as per my example)
>> MiniKuduCluster miniCluster =
>> new
>> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
>>
>>
>> What this would mean:
>>
>> - no change to the existing Kudu code packkaging
>> - leverage caching of the binaries as any Java artifact
>> - download would be at build time not test runtime - only copying out from
>> the jar each time
>> - simple to include in most build environments (not tied to maven as a
>> plugin)
>>
>> I can't comment if it makes sense to do this WRT the binaries though and it
>> would mean building and releasing binaries into a jar on each Kudu release
>> (as protoc does).
>>
>> WDYT?
>>
>> Thanks,
>> Tim
>>
>> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
>> [2]
>>
>> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
>>
>>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org> wrote:
>>>
>>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
>>>
>>> I was imagining that we would try to implement the downloading part as a
>>> Maven plugin, but maybe it could work to try to download the artifacts at
>>> runtime with a JUnit test. Do you think we could cache the artifacts
>>> somewhere, maybe in the Maven repo somehow, so we don't have to download
>>> the artifact every time we want to use it? I was hoping to simply be able
>>> to ship a tarball or a jar of the binaries separately from the test
>>> framework code.
>>>
>>> Mike
>>>
>>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <ti...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks
>>>>
>>>> I've not had too much time, but I threw this together using Mike's
>>>> binaries:
>>>> https://github.com/timrobertson100/kudu-test-server
>>>>
>>>> I think this shows that running a mini cluster is possible using the
>>>> binaries Mike prepared when they are included in a jar (on CentOS 7.4
>> at
>>>> least).
>>>>
>>>> Please don't flame me for the code - it was a rush job - but perhaps
>> you
>>>> could verify it works for you Mike?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org>
>> wrote:
>>>>
>>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
>>> timrobertson100@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>>> I'm pretty busy this week and would be happy to get some help on
>>> this
>>>>> if
>>>>>> anybody has cycles to spare.
>>>>>>
>>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy
>> to
>>>>> offer
>>>>>> a PR for 4).
>>>>>> This is an evening project for me so I might be a little slow too -
>>> if
>>>>>> someone else is keen and has time please feel free to jump in.
>>>>>>
>>>>>
>>>>> Sounds great! I think Grant started working on #4 but I don't know
>> how
>>>> far
>>>>> he got.
>>>>>
>>>>> Mike
>>>>>
>>>>
>>>
>>
Re: Binaries for embedded testing
Posted by Mike Percy <mp...@apache.org>.
Hi Tim,
Sorry for the delay in responding, I've been trying to get back to this.
You make some great points. If we can forego Maven/Gradle plugins, we can
do this with a lot less work. Initially, I was concerned about unpacking
the bits potentially more than strictly necessary, but it seems likely that
the CPU / IO required to do the unpacking would probably not be noticeable
in the grand scheme of running tests against a Kudu MiniCluster, especially
if it's only done once per test file (using @BeforeClass or similar). Also,
as long as there is some way to clean up the files that were unpacked and
avoid potentially filling up a temp dir then nobody should have a problem
with this approach... perhaps we can register a JVM shutdown hook and also
provide some kind of teardown() method so people can ensure the files get
cleaned up as appropriate (I saw the TODO in your test code for this;
protoc uses File.deleteOnExit()).
Regarding creating the binary artifacts, I don't think it's too big of a
burden to ask the release manager to upload either a RHEL 6 or macOS binary
test artifact based on the output of a script, and ask the PMC for help to
get a binary for the other platform.
I'm out of town for the next couple of weeks but once I get back I'll see
if I can push this forward a bit more.
Thanks!
Mike
On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <ti...@gmail.com>
wrote:
> Thanks to you for making this test possible Mike.
>
> I was approaching this emulating protoc where they put the binaries as
> artifacts for Maven [1].
> A slight difference is that protoc is a single file while your tarball has
> several. I notice that the protoc build plugin for maven also copies out
> from the jar to a local filesystem [2] so I just copied that approach.
>
> What I could imagine is we have a jar with the binaries and a single class
> (say EmbeddedKudu) that copies the binaries into a temporary directory (as
> my hack in GH).
>
> A user would then do the following:
>
> <!-- finds the environment -->
> <build>
> <extensions>
> <extension>
> <groupId>kr.motd.maven</groupId>
> <artifactId>os-maven-plugin</artifactId>
> <version>1.6.0</version>
> </extension>
> </extensions>
> </build>
> ....
> <!-- Adds mini cluster -->
> <dependency>
> <groupId>org.apache.kudu</groupId>
> <artifactId>kudu-client</artifactId>
> <version>1.7.0</version>
> <classifier>tests</classifier>
> </dependency>
> <!-- Adds an embedded Kudu -->
> <dependency>
> <groupId>org.apache.kudu</groupId>
> <artifactId>kudu-test-server</artifactId>
> <version>1.7.0</version>
> <classifier>${os.detected.classifier}</classifier>
> </dependency>
>
>
> The detect classifier stuff is copied from protoc, but perhaps we'd just
> state that available options are (?) linux / OSX and ignore autodetection.
>
> And in code users would do something like:
>
> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
> system variable (as per my example)
> MiniKuduCluster miniCluster =
> new
> MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
>
>
> What this would mean:
>
> - no change to the existing Kudu code packkaging
> - leverage caching of the binaries as any Java artifact
> - download would be at build time not test runtime - only copying out from
> the jar each time
> - simple to include in most build environments (not tied to maven as a
> plugin)
>
> I can't comment if it makes sense to do this WRT the binaries though and it
> would mean building and releasing binaries into a jar on each Kudu release
> (as protoc does).
>
> WDYT?
>
> Thanks,
> Tim
>
> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
> [2]
>
> https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
>
> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org> wrote:
>
> > Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
> >
> > I was imagining that we would try to implement the downloading part as a
> > Maven plugin, but maybe it could work to try to download the artifacts at
> > runtime with a JUnit test. Do you think we could cache the artifacts
> > somewhere, maybe in the Maven repo somehow, so we don't have to download
> > the artifact every time we want to use it? I was hoping to simply be able
> > to ship a tarball or a jar of the binaries separately from the test
> > framework code.
> >
> > Mike
> >
> > On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <ti...@gmail.com>
> > wrote:
> >
> > > Hi folks
> > >
> > > I've not had too much time, but I threw this together using Mike's
> > > binaries:
> > > https://github.com/timrobertson100/kudu-test-server
> > >
> > > I think this shows that running a mini cluster is possible using the
> > > binaries Mike prepared when they are included in a jar (on CentOS 7.4
> at
> > > least).
> > >
> > > Please don't flame me for the code - it was a rush job - but perhaps
> you
> > > could verify it works for you Mike?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org>
> wrote:
> > >
> > > > On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
> > timrobertson100@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > > I'm pretty busy this week and would be happy to get some help on
> > this
> > > > if
> > > > > anybody has cycles to spare.
> > > > >
> > > > > Thanks Mike - I'll look into 3) and if we get it working I'm happy
> to
> > > > offer
> > > > > a PR for 4).
> > > > > This is an evening project for me so I might be a little slow too -
> > if
> > > > > someone else is keen and has time please feel free to jump in.
> > > > >
> > > >
> > > > Sounds great! I think Grant started working on #4 but I don't know
> how
> > > far
> > > > he got.
> > > >
> > > > Mike
> > > >
> > >
> >
>
Re: Binaries for embedded testing
Posted by Tim Robertson <ti...@gmail.com>.
Thanks to you for making this test possible Mike.
I was approaching this emulating protoc where they put the binaries as
artifacts for Maven [1].
A slight difference is that protoc is a single file while your tarball has
several. I notice that the protoc build plugin for maven also copies out
from the jar to a local filesystem [2] so I just copied that approach.
What I could imagine is we have a jar with the binaries and a single class
(say EmbeddedKudu) that copies the binaries into a temporary directory (as
my hack in GH).
A user would then do the following:
<!-- finds the environment -->
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.6.0</version>
</extension>
</extensions>
</build>
....
<!-- Adds mini cluster -->
<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-client</artifactId>
<version>1.7.0</version>
<classifier>tests</classifier>
</dependency>
<!-- Adds an embedded Kudu -->
<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-test-server</artifactId>
<version>1.7.0</version>
<classifier>${os.detected.classifier}</classifier>
</dependency>
The detect classifier stuff is copied from protoc, but perhaps we'd just
state that available options are (?) linux / OSX and ignore autodetection.
And in code users would do something like:
EmbeddedKudu.prepare(); // copies kudu from the jar and sets the
system variable (as per my example)
MiniKuduCluster miniCluster =
new MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build();
What this would mean:
- no change to the existing Kudu code packkaging
- leverage caching of the binaries as any Java artifact
- download would be at build time not test runtime - only copying out from
the jar each time
- simple to include in most build environments (not tied to maven as a
plugin)
I can't comment if it makes sense to do this WRT the binaries though and it
would mean building and releasing binaries into a jar on each Kudu release
(as protoc does).
WDYT?
Thanks,
Tim
[1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/
[2]
https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664
On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <mp...@apache.org> wrote:
> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
>
> I was imagining that we would try to implement the downloading part as a
> Maven plugin, but maybe it could work to try to download the artifacts at
> runtime with a JUnit test. Do you think we could cache the artifacts
> somewhere, maybe in the Maven repo somehow, so we don't have to download
> the artifact every time we want to use it? I was hoping to simply be able
> to ship a tarball or a jar of the binaries separately from the test
> framework code.
>
> Mike
>
> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <ti...@gmail.com>
> wrote:
>
> > Hi folks
> >
> > I've not had too much time, but I threw this together using Mike's
> > binaries:
> > https://github.com/timrobertson100/kudu-test-server
> >
> > I think this shows that running a mini cluster is possible using the
> > binaries Mike prepared when they are included in a jar (on CentOS 7.4 at
> > least).
> >
> > Please don't flame me for the code - it was a rush job - but perhaps you
> > could verify it works for you Mike?
> >
> >
> >
> >
> >
> >
> > On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org> wrote:
> >
> > > On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <
> timrobertson100@gmail.com
> > >
> > > wrote:
> > >
> > > > > I'm pretty busy this week and would be happy to get some help on
> this
> > > if
> > > > anybody has cycles to spare.
> > > >
> > > > Thanks Mike - I'll look into 3) and if we get it working I'm happy to
> > > offer
> > > > a PR for 4).
> > > > This is an evening project for me so I might be a little slow too -
> if
> > > > someone else is keen and has time please feel free to jump in.
> > > >
> > >
> > > Sounds great! I think Grant started working on #4 but I don't know how
> > far
> > > he got.
> > >
> > > Mike
> > >
> >
>
Re: Binaries for embedded testing
Posted by Mike Percy <mp...@apache.org>.
Ha, neat, thanks for posting this, Tim. It's a nice proof of concept.
I was imagining that we would try to implement the downloading part as a
Maven plugin, but maybe it could work to try to download the artifacts at
runtime with a JUnit test. Do you think we could cache the artifacts
somewhere, maybe in the Maven repo somehow, so we don't have to download
the artifact every time we want to use it? I was hoping to simply be able
to ship a tarball or a jar of the binaries separately from the test
framework code.
Mike
On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <ti...@gmail.com>
wrote:
> Hi folks
>
> I've not had too much time, but I threw this together using Mike's
> binaries:
> https://github.com/timrobertson100/kudu-test-server
>
> I think this shows that running a mini cluster is possible using the
> binaries Mike prepared when they are included in a jar (on CentOS 7.4 at
> least).
>
> Please don't flame me for the code - it was a rush job - but perhaps you
> could verify it works for you Mike?
>
>
>
>
>
>
> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <mp...@apache.org> wrote:
>
> > On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson <timrobertson100@gmail.com
> >
> > wrote:
> >
> > > > I'm pretty busy this week and would be happy to get some help on this
> > if
> > > anybody has cycles to spare.
> > >
> > > Thanks Mike - I'll look into 3) and if we get it working I'm happy to
> > offer
> > > a PR for 4).
> > > This is an evening project for me so I might be a little slow too - if
> > > someone else is keen and has time please feel free to jump in.
> > >
> >
> > Sounds great! I think Grant started working on #4 but I don't know how
> far
> > he got.
> >
> > Mike
> >
>