You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by Mike Percy <mp...@apache.org> on 2019/03/05 08:16:16 UTC
Kudu server binary test jars for 1.9.0
Hi devs,
For the 1.9.0 release, we (Grant Henke, Brian McDevitt, and I) implemented
the capability to start a Kudu minicluster from a JVM environment without
having to first build Kudu locally (work tracked as KUDU-2411
<https://issues.apache.org/jira/browse/KUDU-2411>). This was made possible
by uploading relocatable Kudu binaries (Linux and Mac) to Maven central,
exposing the Kudu MiniCluster libraries for use by external projects, and
adding logic to the MiniCluster to search the classpath for the Kudu
binaries.
The completion of this work landed a little late in the 1.9.0 release
cycle, just before we started creating release candidates, and as a result
the binaries didn't make it into the RC process for a vote. I propose we
release binary convenience artifacts based on the source tag without going
through another vote because (1) they were built with the 1.9.0 RC2 source
code as voted upon, (2) they are provided simply for convenience and are
not needed to use Kudu, and (3) they are currently considered experimental.
I have already audited the binary artifacts for licensing issues and put in
place automation
<https://github.com/apache/kudu/blob/1.9.0-RC2/build-support/mini-cluster/check-license.pl>
for
the license checks at artifact build time. README, NOTICE, and LICENSE are
all included.
If you're interested in these artifacts and would like to see if they work
for you, or would like to take a closer look for other reasons, I've
uploaded binary jar artifacts for Linux and Mac (h/t to Alexey for the El
Capitan build) to the Maven staging repository at
https://repository.apache.org/content/repositories/orgapachekudu-1033/
We also have an example of using the above binary artifacts, built from the
1.9.0-RC2 tag and using 1.9.0-RC2 Java client libraries at
https://github.com/mpercy/kudu-binary-jar-test/ -- you can see if the
artifact works in your environment by simply cloning that Git repository
and running "mvn test". (This is just a copy/pasted and modified version of
java-example from the Kudu code examples repo; h/t to Brian for getting it
to work with the binary jar.)
Please note that the binary jar Maven integration (using os-maven-plugin)
requires Maven 3.1 or later. In addition, it must be run on macOS El
Capitan or later, or a modern flavor of Linux (any Linux distribution that
Kudu supports
<https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements>
should work).
A draft of user docs for how to integrate the Kudu binary jar into a Java
project can be found here
<https://docs.google.com/document/d/1n9flFdHEnULX3_wxmy__gU7WKK-FIYFxvufWbvEHu-c/>
and
if we have consensus on including this as outlined above then I'll work
with Andrew to get the relevant docs posted to the web site in time for the
1.9.0 release announcement.
Thanks to those who helped get this done (some early prototyping was also
done by Tim Robertson) and I hope that folks find this new capability
useful!
Please let me know if you have questions, concerns, or feedback on this
functionality.
Regards,
Mike
Re: Kudu server binary test jars for 1.9.0
Posted by Mike Percy <mp...@apache.org>.
Tim: Sounds good!
Grant: Thanks for the broad testing! Let me know if you think we need to
additionally document anything.
Andrew: Thanks for testing on High Sierra and for being accommodating on
this!
Mike
On Tue, Mar 5, 2019 at 10:26 AM Andrew Wong <aw...@apache.org> wrote:
> I ran your example on macOS High Sierra 10.13.6 and on CentOS 7.3, only
> issue I ran into was because of the Maven version.
> I'll let this sit for the day or so in case people want to chyme in, and
> then proceed with the release.
>
> Thanks for pushing this through!
>
> On Tue, Mar 5, 2019 at 8:44 AM Grant Henke <gh...@cloudera.com.invalid>
> wrote:
>
> > +1 on this plan. Thanks for wrapping this up Mike!
> >
> > I ran the test locally on my Mac (10.14.2) with no issues.
> >
> > I also ran the tests in a docker image for the following operating
> systems:
> >
> > - centos:6
> > - centos:7
> > - debian:jessie
> > - debian:stretch
> > - ubuntu:trusty
> > - ubuntu:xenial
> > - ubuntu:bionic
> >
> > I ran into a few hiccups during testing, but they were all easy to
> resolve:
> >
> > - I needed to upgrade the default Maven version on ubuntu:trusty and
> > centos.
> > - I ran into a open-jdk/maven-surefire bug on debian:stretch
> > -
> >
> >
> https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class
> > - Fixed with:
> >
> >
> > >
> > >
> > >
> > >
> > >
> > > *<plugin> <groupId>org.apache.maven.plugins</groupId>
> > > <artifactId>maven-surefire-plugin</artifactId> <configuration>
> > > <useSystemClassLoader>false</useSystemClassLoader>
> > > </configuration></plugin>*
> >
> >
> > The setup varies a bit based on the OS but my basic test script was
> > effectively this:
> >
> > *docker run -it --rm --privileged ubuntu:xenial /bin/bash*
> >
> >
> > > *apt-get update -y*
> >
> > *apt-get install -y git maven ntp openjdk-8-jdk*
> >
> >
> > > *echo "listen on *" >> /etc/ntp.conf*
> >
> > *service ntp restart*
> >
> >
> > > *git clone https://github.com/mpercy/kudu-binary-jar-test/
> > > <https://github.com/mpercy/kudu-binary-jar-test/>*
> >
> > *cd kudu-binary-jar-test*
> >
> > *mvn test*
> >
> >
> >
> >
> >
> > On Tue, Mar 5, 2019 at 2:57 AM Tim Robertson <ti...@gmail.com>
> > wrote:
> >
> > > Thank you for progressing all this Mike (and others).
> > > I will endeavour to test this soon and replace the Beam unit tests with
> > > this - I've been sidelined a bit recently with other tasks but getting
> > back
> > > into Beam development again.
> > >
> > >
> > >
> > >
> > > On Tue, Mar 5, 2019 at 9:16 AM Mike Percy <mp...@apache.org> wrote:
> > >
> > > > Hi devs,
> > > > For the 1.9.0 release, we (Grant Henke, Brian McDevitt, and I)
> > > implemented
> > > > the capability to start a Kudu minicluster from a JVM environment
> > without
> > > > having to first build Kudu locally (work tracked as KUDU-2411
> > > > <https://issues.apache.org/jira/browse/KUDU-2411>). This was made
> > > possible
> > > > by uploading relocatable Kudu binaries (Linux and Mac) to Maven
> > central,
> > > > exposing the Kudu MiniCluster libraries for use by external projects,
> > and
> > > > adding logic to the MiniCluster to search the classpath for the Kudu
> > > > binaries.
> > > >
> > > > The completion of this work landed a little late in the 1.9.0 release
> > > > cycle, just before we started creating release candidates, and as a
> > > result
> > > > the binaries didn't make it into the RC process for a vote. I propose
> > we
> > > > release binary convenience artifacts based on the source tag without
> > > going
> > > > through another vote because (1) they were built with the 1.9.0 RC2
> > > source
> > > > code as voted upon, (2) they are provided simply for convenience and
> > are
> > > > not needed to use Kudu, and (3) they are currently considered
> > > experimental.
> > > >
> > > > I have already audited the binary artifacts for licensing issues and
> > put
> > > in
> > > > place automation
> > > > <
> > > >
> > >
> >
> https://github.com/apache/kudu/blob/1.9.0-RC2/build-support/mini-cluster/check-license.pl
> > > > >
> > > > for
> > > > the license checks at artifact build time. README, NOTICE, and
> LICENSE
> > > are
> > > > all included.
> > > >
> > > > If you're interested in these artifacts and would like to see if they
> > > work
> > > > for you, or would like to take a closer look for other reasons, I've
> > > > uploaded binary jar artifacts for Linux and Mac (h/t to Alexey for
> the
> > El
> > > > Capitan build) to the Maven staging repository at
> > > >
> https://repository.apache.org/content/repositories/orgapachekudu-1033/
> > > >
> > > > We also have an example of using the above binary artifacts, built
> from
> > > the
> > > > 1.9.0-RC2 tag and using 1.9.0-RC2 Java client libraries at
> > > > https://github.com/mpercy/kudu-binary-jar-test/ -- you can see if
> the
> > > > artifact works in your environment by simply cloning that Git
> > repository
> > > > and running "mvn test". (This is just a copy/pasted and modified
> > version
> > > of
> > > > java-example from the Kudu code examples repo; h/t to Brian for
> getting
> > > it
> > > > to work with the binary jar.)
> > > >
> > > > Please note that the binary jar Maven integration (using
> > os-maven-plugin)
> > > > requires Maven 3.1 or later. In addition, it must be run on macOS El
> > > > Capitan or later, or a modern flavor of Linux (any Linux distribution
> > > that
> > > > Kudu supports
> > > > <
> > > >
> > >
> >
> https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements
> > > > >
> > > > should work).
> > > >
> > > > A draft of user docs for how to integrate the Kudu binary jar into a
> > Java
> > > > project can be found here
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1n9flFdHEnULX3_wxmy__gU7WKK-FIYFxvufWbvEHu-c/
> > > > >
> > > > and
> > > > if we have consensus on including this as outlined above then I'll
> work
> > > > with Andrew to get the relevant docs posted to the web site in time
> for
> > > the
> > > > 1.9.0 release announcement.
> > > >
> > > > Thanks to those who helped get this done (some early prototyping was
> > also
> > > > done by Tim Robertson) and I hope that folks find this new capability
> > > > useful!
> > > >
> > > > Please let me know if you have questions, concerns, or feedback on
> this
> > > > functionality.
> > > >
> > > > Regards,
> > > > Mike
> > > >
> > >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
Re: Kudu server binary test jars for 1.9.0
Posted by Andrew Wong <aw...@apache.org>.
I ran your example on macOS High Sierra 10.13.6 and on CentOS 7.3, only
issue I ran into was because of the Maven version.
I'll let this sit for the day or so in case people want to chyme in, and
then proceed with the release.
Thanks for pushing this through!
On Tue, Mar 5, 2019 at 8:44 AM Grant Henke <gh...@cloudera.com.invalid>
wrote:
> +1 on this plan. Thanks for wrapping this up Mike!
>
> I ran the test locally on my Mac (10.14.2) with no issues.
>
> I also ran the tests in a docker image for the following operating systems:
>
> - centos:6
> - centos:7
> - debian:jessie
> - debian:stretch
> - ubuntu:trusty
> - ubuntu:xenial
> - ubuntu:bionic
>
> I ran into a few hiccups during testing, but they were all easy to resolve:
>
> - I needed to upgrade the default Maven version on ubuntu:trusty and
> centos.
> - I ran into a open-jdk/maven-surefire bug on debian:stretch
> -
>
> https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class
> - Fixed with:
>
>
> >
> >
> >
> >
> >
> > *<plugin> <groupId>org.apache.maven.plugins</groupId>
> > <artifactId>maven-surefire-plugin</artifactId> <configuration>
> > <useSystemClassLoader>false</useSystemClassLoader>
> > </configuration></plugin>*
>
>
> The setup varies a bit based on the OS but my basic test script was
> effectively this:
>
> *docker run -it --rm --privileged ubuntu:xenial /bin/bash*
>
>
> > *apt-get update -y*
>
> *apt-get install -y git maven ntp openjdk-8-jdk*
>
>
> > *echo "listen on *" >> /etc/ntp.conf*
>
> *service ntp restart*
>
>
> > *git clone https://github.com/mpercy/kudu-binary-jar-test/
> > <https://github.com/mpercy/kudu-binary-jar-test/>*
>
> *cd kudu-binary-jar-test*
>
> *mvn test*
>
>
>
>
>
> On Tue, Mar 5, 2019 at 2:57 AM Tim Robertson <ti...@gmail.com>
> wrote:
>
> > Thank you for progressing all this Mike (and others).
> > I will endeavour to test this soon and replace the Beam unit tests with
> > this - I've been sidelined a bit recently with other tasks but getting
> back
> > into Beam development again.
> >
> >
> >
> >
> > On Tue, Mar 5, 2019 at 9:16 AM Mike Percy <mp...@apache.org> wrote:
> >
> > > Hi devs,
> > > For the 1.9.0 release, we (Grant Henke, Brian McDevitt, and I)
> > implemented
> > > the capability to start a Kudu minicluster from a JVM environment
> without
> > > having to first build Kudu locally (work tracked as KUDU-2411
> > > <https://issues.apache.org/jira/browse/KUDU-2411>). This was made
> > possible
> > > by uploading relocatable Kudu binaries (Linux and Mac) to Maven
> central,
> > > exposing the Kudu MiniCluster libraries for use by external projects,
> and
> > > adding logic to the MiniCluster to search the classpath for the Kudu
> > > binaries.
> > >
> > > The completion of this work landed a little late in the 1.9.0 release
> > > cycle, just before we started creating release candidates, and as a
> > result
> > > the binaries didn't make it into the RC process for a vote. I propose
> we
> > > release binary convenience artifacts based on the source tag without
> > going
> > > through another vote because (1) they were built with the 1.9.0 RC2
> > source
> > > code as voted upon, (2) they are provided simply for convenience and
> are
> > > not needed to use Kudu, and (3) they are currently considered
> > experimental.
> > >
> > > I have already audited the binary artifacts for licensing issues and
> put
> > in
> > > place automation
> > > <
> > >
> >
> https://github.com/apache/kudu/blob/1.9.0-RC2/build-support/mini-cluster/check-license.pl
> > > >
> > > for
> > > the license checks at artifact build time. README, NOTICE, and LICENSE
> > are
> > > all included.
> > >
> > > If you're interested in these artifacts and would like to see if they
> > work
> > > for you, or would like to take a closer look for other reasons, I've
> > > uploaded binary jar artifacts for Linux and Mac (h/t to Alexey for the
> El
> > > Capitan build) to the Maven staging repository at
> > > https://repository.apache.org/content/repositories/orgapachekudu-1033/
> > >
> > > We also have an example of using the above binary artifacts, built from
> > the
> > > 1.9.0-RC2 tag and using 1.9.0-RC2 Java client libraries at
> > > https://github.com/mpercy/kudu-binary-jar-test/ -- you can see if the
> > > artifact works in your environment by simply cloning that Git
> repository
> > > and running "mvn test". (This is just a copy/pasted and modified
> version
> > of
> > > java-example from the Kudu code examples repo; h/t to Brian for getting
> > it
> > > to work with the binary jar.)
> > >
> > > Please note that the binary jar Maven integration (using
> os-maven-plugin)
> > > requires Maven 3.1 or later. In addition, it must be run on macOS El
> > > Capitan or later, or a modern flavor of Linux (any Linux distribution
> > that
> > > Kudu supports
> > > <
> > >
> >
> https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements
> > > >
> > > should work).
> > >
> > > A draft of user docs for how to integrate the Kudu binary jar into a
> Java
> > > project can be found here
> > > <
> > >
> >
> https://docs.google.com/document/d/1n9flFdHEnULX3_wxmy__gU7WKK-FIYFxvufWbvEHu-c/
> > > >
> > > and
> > > if we have consensus on including this as outlined above then I'll work
> > > with Andrew to get the relevant docs posted to the web site in time for
> > the
> > > 1.9.0 release announcement.
> > >
> > > Thanks to those who helped get this done (some early prototyping was
> also
> > > done by Tim Robertson) and I hope that folks find this new capability
> > > useful!
> > >
> > > Please let me know if you have questions, concerns, or feedback on this
> > > functionality.
> > >
> > > Regards,
> > > Mike
> > >
> >
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>
Re: Kudu server binary test jars for 1.9.0
Posted by Grant Henke <gh...@cloudera.com.INVALID>.
+1 on this plan. Thanks for wrapping this up Mike!
I ran the test locally on my Mac (10.14.2) with no issues.
I also ran the tests in a docker image for the following operating systems:
- centos:6
- centos:7
- debian:jessie
- debian:stretch
- ubuntu:trusty
- ubuntu:xenial
- ubuntu:bionic
I ran into a few hiccups during testing, but they were all easy to resolve:
- I needed to upgrade the default Maven version on ubuntu:trusty and
centos.
- I ran into a open-jdk/maven-surefire bug on debian:stretch
-
https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class
- Fixed with:
>
>
>
>
>
> *<plugin> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId> <configuration>
> <useSystemClassLoader>false</useSystemClassLoader>
> </configuration></plugin>*
The setup varies a bit based on the OS but my basic test script was
effectively this:
*docker run -it --rm --privileged ubuntu:xenial /bin/bash*
> *apt-get update -y*
*apt-get install -y git maven ntp openjdk-8-jdk*
> *echo "listen on *" >> /etc/ntp.conf*
*service ntp restart*
> *git clone https://github.com/mpercy/kudu-binary-jar-test/
> <https://github.com/mpercy/kudu-binary-jar-test/>*
*cd kudu-binary-jar-test*
*mvn test*
On Tue, Mar 5, 2019 at 2:57 AM Tim Robertson <ti...@gmail.com>
wrote:
> Thank you for progressing all this Mike (and others).
> I will endeavour to test this soon and replace the Beam unit tests with
> this - I've been sidelined a bit recently with other tasks but getting back
> into Beam development again.
>
>
>
>
> On Tue, Mar 5, 2019 at 9:16 AM Mike Percy <mp...@apache.org> wrote:
>
> > Hi devs,
> > For the 1.9.0 release, we (Grant Henke, Brian McDevitt, and I)
> implemented
> > the capability to start a Kudu minicluster from a JVM environment without
> > having to first build Kudu locally (work tracked as KUDU-2411
> > <https://issues.apache.org/jira/browse/KUDU-2411>). This was made
> possible
> > by uploading relocatable Kudu binaries (Linux and Mac) to Maven central,
> > exposing the Kudu MiniCluster libraries for use by external projects, and
> > adding logic to the MiniCluster to search the classpath for the Kudu
> > binaries.
> >
> > The completion of this work landed a little late in the 1.9.0 release
> > cycle, just before we started creating release candidates, and as a
> result
> > the binaries didn't make it into the RC process for a vote. I propose we
> > release binary convenience artifacts based on the source tag without
> going
> > through another vote because (1) they were built with the 1.9.0 RC2
> source
> > code as voted upon, (2) they are provided simply for convenience and are
> > not needed to use Kudu, and (3) they are currently considered
> experimental.
> >
> > I have already audited the binary artifacts for licensing issues and put
> in
> > place automation
> > <
> >
> https://github.com/apache/kudu/blob/1.9.0-RC2/build-support/mini-cluster/check-license.pl
> > >
> > for
> > the license checks at artifact build time. README, NOTICE, and LICENSE
> are
> > all included.
> >
> > If you're interested in these artifacts and would like to see if they
> work
> > for you, or would like to take a closer look for other reasons, I've
> > uploaded binary jar artifacts for Linux and Mac (h/t to Alexey for the El
> > Capitan build) to the Maven staging repository at
> > https://repository.apache.org/content/repositories/orgapachekudu-1033/
> >
> > We also have an example of using the above binary artifacts, built from
> the
> > 1.9.0-RC2 tag and using 1.9.0-RC2 Java client libraries at
> > https://github.com/mpercy/kudu-binary-jar-test/ -- you can see if the
> > artifact works in your environment by simply cloning that Git repository
> > and running "mvn test". (This is just a copy/pasted and modified version
> of
> > java-example from the Kudu code examples repo; h/t to Brian for getting
> it
> > to work with the binary jar.)
> >
> > Please note that the binary jar Maven integration (using os-maven-plugin)
> > requires Maven 3.1 or later. In addition, it must be run on macOS El
> > Capitan or later, or a modern flavor of Linux (any Linux distribution
> that
> > Kudu supports
> > <
> >
> https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements
> > >
> > should work).
> >
> > A draft of user docs for how to integrate the Kudu binary jar into a Java
> > project can be found here
> > <
> >
> https://docs.google.com/document/d/1n9flFdHEnULX3_wxmy__gU7WKK-FIYFxvufWbvEHu-c/
> > >
> > and
> > if we have consensus on including this as outlined above then I'll work
> > with Andrew to get the relevant docs posted to the web site in time for
> the
> > 1.9.0 release announcement.
> >
> > Thanks to those who helped get this done (some early prototyping was also
> > done by Tim Robertson) and I hope that folks find this new capability
> > useful!
> >
> > Please let me know if you have questions, concerns, or feedback on this
> > functionality.
> >
> > Regards,
> > Mike
> >
>
--
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: Kudu server binary test jars for 1.9.0
Posted by Tim Robertson <ti...@gmail.com>.
Thank you for progressing all this Mike (and others).
I will endeavour to test this soon and replace the Beam unit tests with
this - I've been sidelined a bit recently with other tasks but getting back
into Beam development again.
On Tue, Mar 5, 2019 at 9:16 AM Mike Percy <mp...@apache.org> wrote:
> Hi devs,
> For the 1.9.0 release, we (Grant Henke, Brian McDevitt, and I) implemented
> the capability to start a Kudu minicluster from a JVM environment without
> having to first build Kudu locally (work tracked as KUDU-2411
> <https://issues.apache.org/jira/browse/KUDU-2411>). This was made possible
> by uploading relocatable Kudu binaries (Linux and Mac) to Maven central,
> exposing the Kudu MiniCluster libraries for use by external projects, and
> adding logic to the MiniCluster to search the classpath for the Kudu
> binaries.
>
> The completion of this work landed a little late in the 1.9.0 release
> cycle, just before we started creating release candidates, and as a result
> the binaries didn't make it into the RC process for a vote. I propose we
> release binary convenience artifacts based on the source tag without going
> through another vote because (1) they were built with the 1.9.0 RC2 source
> code as voted upon, (2) they are provided simply for convenience and are
> not needed to use Kudu, and (3) they are currently considered experimental.
>
> I have already audited the binary artifacts for licensing issues and put in
> place automation
> <
> https://github.com/apache/kudu/blob/1.9.0-RC2/build-support/mini-cluster/check-license.pl
> >
> for
> the license checks at artifact build time. README, NOTICE, and LICENSE are
> all included.
>
> If you're interested in these artifacts and would like to see if they work
> for you, or would like to take a closer look for other reasons, I've
> uploaded binary jar artifacts for Linux and Mac (h/t to Alexey for the El
> Capitan build) to the Maven staging repository at
> https://repository.apache.org/content/repositories/orgapachekudu-1033/
>
> We also have an example of using the above binary artifacts, built from the
> 1.9.0-RC2 tag and using 1.9.0-RC2 Java client libraries at
> https://github.com/mpercy/kudu-binary-jar-test/ -- you can see if the
> artifact works in your environment by simply cloning that Git repository
> and running "mvn test". (This is just a copy/pasted and modified version of
> java-example from the Kudu code examples repo; h/t to Brian for getting it
> to work with the binary jar.)
>
> Please note that the binary jar Maven integration (using os-maven-plugin)
> requires Maven 3.1 or later. In addition, it must be run on macOS El
> Capitan or later, or a modern flavor of Linux (any Linux distribution that
> Kudu supports
> <
> https://kudu.apache.org/docs/installation.html#_prerequisites_and_requirements
> >
> should work).
>
> A draft of user docs for how to integrate the Kudu binary jar into a Java
> project can be found here
> <
> https://docs.google.com/document/d/1n9flFdHEnULX3_wxmy__gU7WKK-FIYFxvufWbvEHu-c/
> >
> and
> if we have consensus on including this as outlined above then I'll work
> with Andrew to get the relevant docs posted to the web site in time for the
> 1.9.0 release announcement.
>
> Thanks to those who helped get this done (some early prototyping was also
> done by Tim Robertson) and I hope that folks find this new capability
> useful!
>
> Please let me know if you have questions, concerns, or feedback on this
> functionality.
>
> Regards,
> Mike
>