Re: [Migration Help]

[leerho <le...@gmail.com>]

Gian,
Thanks!


> https://github.com/apache/incubator-druid/pull/6482.


The discussion about the traceability of the GA back to the final RC is
interesting.
I gather you decided not to have a git.version file at the root and you
added that info into the jar manifests, so where do you have that
information for the tar.gz assemblies?

Also, it is more work upfront, but you might find that in the long run, you
> enjoy switching the project to a single repo with a multi-module Maven
> build. We do this in Druid and it works quite well. It makes it way easier
> to do patches and releases that span multiple modules.


The DataSketches library is not like a single product with components that
are released together.  Some of our "components" have different life-cycles
and versioning.  Also, we have C++ / Python repositories that don't fit
well with Maven, which is all about Java.

Even the Java-only components have vastly different dependencies and
version cycles and don't necessarily need to be released at the same time
the core-java component is released.  So far, we don't have releases that
span multiple components.  For example, the core-java library may be
several versions ahead of the hive or pig components.

Definitely doing as much as possible in the build system (whether Maven or
> Gradle) is a good thing, IMO, since they tend to reward living in their
> worlds. Also the more verifications you can push into the build tool, the
> better.


I agree.

GPG ?


You are correct.  The .asc signature is generated with GPG.




On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org> wrote:

> Hey Lee,
>
> You might find this PR useful, it is where we Apache-fied the Druid POM:
> https://github.com/apache/incubator-druid/pull/6482. It helps highlight
> the
> specific areas that are relevant to Apache-fying a build.
>
> Also, it is more work upfront, but you might find that in the long run, you
> enjoy switching the project to a single repo with a multi-module Maven
> build. We do this in Druid and it works quite well. It makes it way easier
> to do patches and releases that span multiple modules.
>
> > So it seems that eventually, either we need to go whole hog having Maven
> do
> > everything, which seems overly complicated, or consider moving to Gradle
> > (which I am not familiar with either, but it seems lots of teams are
> moving
> > that way).
>
> Definitely doing as much as possible in the build system (whether Maven or
> Gradle) is a good thing, IMO, since they tend to reward living in their
> worlds. Also the more verifications you can push into the build tool, the
> better.
>
> In Druid we have also been considering moving to Gradle:
> https://github.com/apache/incubator-druid/issues/7866. However we are
> still
> using Maven for now and may be for some time to come, since it seems the
> benefits of switching to Gradle are definitely noticeable, but not huge.
>
> > - It doesn't appear that the Nexus artifacts require a GPG signature only
> > the DIST assembly requires GPG, which the script is handling.  So why do
> we
> > need the gpg-plugin?
>
> IIRC it's because Sonatype's Central Repository (where the Maven-ready
> artifacts ultimately end up) requires GPG signatures.
>
> > - It is not clear to me when a profile should be used (e.g., for deploy)
> vs
> > using the deploy process as part of the pom main build lifecycle.
>
> I've found profiles to be most useful when you want something like "mvn
> package" to do different things based on a profile. e.g. having a profile
> to apply strict checks that takes extra time, having a profile to build
> extra assemblies, etc.
>

Re: [Migration Help]

[leerho <le...@gmail.com>]

Holy Jeez, no wonder I was puzzled :)

On Fri, Aug 2, 2019 at 2:59 PM Gian Merlino <gi...@apache.org> wrote:

> > What is confusing is that the Apache parent pom defines the
> > maven-release-plugin
> > only in build.pluginManagement where it calls the apache-release profile.
> > Similarly, the Druid pom only defines the release plugin in
> > build.pluginManagement.
> > It has been my understanding that "pluginManagement" was only for
> > inheritance.
> > and that for a plugin to be actually executed it had to be referenced in
> > the build.plugins
> > section or in a profile.build.plugins.
>
> This is getting outside of my Maven knowledge, but I would guess that the
> release plugin is special and doesn't need to be listed as an explicit
> plugin, since it has a special command to activate it (mvn release:prepare
> or release:perform). It doesn't run on "normal" Maven commands.
>
> > I don't see a profile called "dist".
> > Where do I find that?
>
> Both list and source-release-assembly-druid are in the "distribution" pom:
> https://github.com/apache/incubator-druid/blob/master/distribution/pom.xml
> .
> (Druid is a multi-module project, and the "distribution" module is the one
> that builds the tarballs. The main pom 'sets the stage' but doesn't
> actually do anything.)
>
> On Fri, Aug 2, 2019 at 2:18 PM leerho <le...@gmail.com> wrote:
>
> > >
> > > It happens as part of the maven-release-plugin.
> >
> > What is confusing is that the Apache parent pom defines the
> > maven-release-plugin
> > only in build.pluginManagement where it calls the apache-release profile.
> > Similarly, the Druid pom only defines the release plugin in
> > build.pluginManagement.
> > It has been my understanding that "pluginManagement" was only for
> > inheritance.
> > and that for a plugin to be actually executed it had to be referenced in
> > the build.plugins
> > section or in a profile.build.plugins.
> >
> >
> >
> > > - Check out the tag, then build release candidate with:
> > > mvn clean install
> > >     -Papache-release,dist,rat
> > >     -DskipTests
> > >     -Dgpg.keyname=<your GPG key fingerprint>
> >
> > I don't see a profile called "dist".
> >
> > We've defined our own, check out "source-release-assembly-druid".
> >
> >
> > Where do I find that?
> >
> >
> >
> >
> >
> >
> > On Fri, Aug 2, 2019 at 12:37 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > > 1) When you deploy the jar files to Nexus, I note that they are
> signed
> > by
> > > > GPG, yet, you do not have a GPG plugin defined in your pom. If you
> are
> > > > depending on the Apache parent pom "apache-release" profile, then
> where
> > > is
> > > > it being called?  From a script?
> > >
> > > It happens as part of the maven-release-plugin. The commands we run to
> > do a
> > > release candidate are something like:
> > >
> > > - Create a tag with: mvn -DreleaseVersion=0.14.1-incubating
> > > -DdevelopmentVersion=0.14.2-incubating-SNAPSHOT
> > > -Dtag=druid-0.14.1-incubating-rc1 -DpushChanges=false clean
> release:clean
> > > release:prepare
> > > - Check out the tag, then build release candidate with: mvn clean
> install
> > > -Papache-release,dist,rat -DskipTests -Dgpg.keyname=<your GPG key
> > > fingerprint>
> > >
> > > Later on, to upload the final artifacts to Nexus:
> > >
> > > - Using a saved release.properties file from (1), run: mvn
> > release:perform
> > > -Darguments=-Dgpg.keyname=<my-key>
> > >
> > > (If you lose the release.properties file, you should be able to recover
> > > anyway by specifying a few more arguments on the release:perform line,
> > but
> > > I don't recall what they are off the top of my head.)
> > >
> > > > 2) You have a profile called "apache-release" that disables the
> > > > source-release-assembly.  For the release to DIST, I presume you are
> > > > building the tarball yourself, where?  The related question is what
> > > script
> > > > is using the above source-assembly.xml file to include the
> git.version
> > +
> > > > other stuff.?
> > >
> > > We've defined our own, check out "source-release-assembly-druid". We
> > > disable the source-release-assembly inherited from the parent pom so we
> > can
> > > use ours instead. It gets built in the "mvn clean install" step above.
> > >
> > > > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > > > documented.  Does this plugin introduce variables into the build
> > process
> > > > like ${git.commit.id}, ${git.build.version}, etc?  They are not
> > defined
> > > > elsewhere.  The jar-plugin needs the to insert into the manifest.
> > >
> > > I don't know the details, but that sounds like a very reasonable guess.
> > >
> > > On Fri, Aug 2, 2019 at 11:46 AM leerho <le...@gmail.com> wrote:
> > >
> > > > Two questions WRT the Druid root pom and deployment process
> > > >
> > > > 1) When you deploy the jar files to Nexus, I note that they are
> signed
> > by
> > > > GPG, yet, you do not have a GPG plugin defined in your pom. If you
> are
> > > > depending on the Apache parent pom "apache-release" profile, then
> where
> > > is
> > > > it being called?  From a script?
> > > >
> > > > 2) You have a profile called "apache-release" that disables the
> > > > source-release-assembly.  For the release to DIST, I presume you are
> > > > building the tarball yourself, where?  The related question is what
> > > script
> > > > is using the above source-assembly.xml file to include the
> git.version
> > +
> > > > other stuff.?
> > > >
> > > > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > > > documented.  Does this plugin introduce variables into the build
> > process
> > > > like ${git.commit.id}, ${git.build.version}, etc?  They are not
> > defined
> > > > elsewhere.  The jar-plugin needs the to insert into the manifest.
> > > >
> > > >
> > > >
> > > > On Thu, Aug 1, 2019 at 11:42 AM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > > OK, I see it now in the tarball.  Where is the script or code
> that
> > > > loads
> > > > > > the data into git.version ?
> > > > >
> > > > > It's done by git-commit-id-plugin in our main POM:
> > > > > https://github.com/apache/incubator-druid/blob/master/pom.xml.
> Then
> > > it's
> > > > > included into the source assembly via:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml
> > > > >
> > > > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just
> > the
> > > > PMC
> > > > > > that votes on a release?
> > > > >
> > > > > Anyone can vote, but only PMC votes are binding. Even top level
> > > projects
> > > > > still should have a 72 hour voting period for each release
> candidate,
> > > > > though, which can make release cycles take longer than you might be
> > > used
> > > > > to.
> > > > >
> > > > > On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:
> > > > >
> > > > > > >
> > > > > > > We do have it in the tarball root for the real release
> > > > > >
> > > > > > OK, I see it now in the tarball.  Where is the script or code
> that
> > > > loads
> > > > > > the data into git.version ?
> > > > > >
> > > > > > If they stay separate...
> > > > > >
> > > > > >
> > > > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just
> > the
> > > > PMC
> > > > > > that votes on a release?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > > I gather you decided not to have a git.version file at the
> root
> > > and
> > > > > you
> > > > > > > > added that info into the jar manifests, so where do you have
> > that
> > > > > > > > information for the tar.gz assemblies?
> > > > > > >
> > > > > > > We do have it in the tarball root for the real release -- the
> > > source
> > > > > > > tarball (ASF policy considers source releases as the sole
> > official
> > > > > > > release). For the binary convenience artifacts, we don't
> include
> > it
> > > > at
> > > > > > the
> > > > > > > root, but the jars have the info if people go digging.
> > > > > > >
> > > > > > > > Even the Java-only components have vastly different
> > dependencies
> > > > and
> > > > > > > > version cycles and don't necessarily need to be released at
> the
> > > > same
> > > > > > time
> > > > > > > > the core-java component is released.  So far, we don't have
> > > > releases
> > > > > > that
> > > > > > > > span multiple components.  For example, the core-java library
> > may
> > > > be
> > > > > > > > several versions ahead of the hive or pig components.
> > > > > > >
> > > > > > > Sure, that makes sense. I would still consider aligning their
> > > release
> > > > > > > processes though. If they stay separate, each component release
> > > would
> > > > > > need
> > > > > > > its own release manager, its own voting cycle, its own publish
> > and
> > > > > > > announcement process, etc, and that can be a lot of
> > administrative
> > > > > > > overhead.
> > > > > > >
> > > > > > > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > > Gian,
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > >
> > > > > > > > > https://github.com/apache/incubator-druid/pull/6482.
> > > > > > > >
> > > > > > > >
> > > > > > > > The discussion about the traceability of the GA back to the
> > final
> > > > RC
> > > > > is
> > > > > > > > interesting.
> > > > > > > > I gather you decided not to have a git.version file at the
> root
> > > and
> > > > > you
> > > > > > > > added that info into the jar manifests, so where do you have
> > that
> > > > > > > > information for the tar.gz assemblies?
> > > > > > > >
> > > > > > > > Also, it is more work upfront, but you might find that in the
> > > long
> > > > > run,
> > > > > > > you
> > > > > > > > > enjoy switching the project to a single repo with a
> > > multi-module
> > > > > > Maven
> > > > > > > > > build. We do this in Druid and it works quite well. It
> makes
> > it
> > > > way
> > > > > > > > easier
> > > > > > > > > to do patches and releases that span multiple modules.
> > > > > > > >
> > > > > > > >
> > > > > > > > The DataSketches library is not like a single product with
> > > > components
> > > > > > > that
> > > > > > > > are released together.  Some of our "components" have
> different
> > > > > > > life-cycles
> > > > > > > > and versioning.  Also, we have C++ / Python repositories that
> > > don't
> > > > > fit
> > > > > > > > well with Maven, which is all about Java.
> > > > > > > >
> > > > > > > > Even the Java-only components have vastly different
> > dependencies
> > > > and
> > > > > > > > version cycles and don't necessarily need to be released at
> the
> > > > same
> > > > > > time
> > > > > > > > the core-java component is released.  So far, we don't have
> > > > releases
> > > > > > that
> > > > > > > > span multiple components.  For example, the core-java library
> > may
> > > > be
> > > > > > > > several versions ahead of the hive or pig components.
> > > > > > > >
> > > > > > > > Definitely doing as much as possible in the build system
> > (whether
> > > > > Maven
> > > > > > > or
> > > > > > > > > Gradle) is a good thing, IMO, since they tend to reward
> > living
> > > in
> > > > > > their
> > > > > > > > > worlds. Also the more verifications you can push into the
> > build
> > > > > tool,
> > > > > > > the
> > > > > > > > > better.
> > > > > > > >
> > > > > > > >
> > > > > > > > I agree.
> > > > > > > >
> > > > > > > > GPG ?
> > > > > > > >
> > > > > > > >
> > > > > > > > You are correct.  The .asc signature is generated with GPG.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <
> gian@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Lee,
> > > > > > > > >
> > > > > > > > > You might find this PR useful, it is where we Apache-fied
> the
> > > > Druid
> > > > > > > POM:
> > > > > > > > > https://github.com/apache/incubator-druid/pull/6482. It
> > helps
> > > > > > > highlight
> > > > > > > > > the
> > > > > > > > > specific areas that are relevant to Apache-fying a build.
> > > > > > > > >
> > > > > > > > > Also, it is more work upfront, but you might find that in
> the
> > > > long
> > > > > > run,
> > > > > > > > you
> > > > > > > > > enjoy switching the project to a single repo with a
> > > multi-module
> > > > > > Maven
> > > > > > > > > build. We do this in Druid and it works quite well. It
> makes
> > it
> > > > way
> > > > > > > > easier
> > > > > > > > > to do patches and releases that span multiple modules.
> > > > > > > > >
> > > > > > > > > > So it seems that eventually, either we need to go whole
> hog
> > > > > having
> > > > > > > > Maven
> > > > > > > > > do
> > > > > > > > > > everything, which seems overly complicated, or consider
> > > moving
> > > > to
> > > > > > > > Gradle
> > > > > > > > > > (which I am not familiar with either, but it seems lots
> of
> > > > teams
> > > > > > are
> > > > > > > > > moving
> > > > > > > > > > that way).
> > > > > > > > >
> > > > > > > > > Definitely doing as much as possible in the build system
> > > (whether
> > > > > > Maven
> > > > > > > > or
> > > > > > > > > Gradle) is a good thing, IMO, since they tend to reward
> > living
> > > in
> > > > > > their
> > > > > > > > > worlds. Also the more verifications you can push into the
> > build
> > > > > tool,
> > > > > > > the
> > > > > > > > > better.
> > > > > > > > >
> > > > > > > > > In Druid we have also been considering moving to Gradle:
> > > > > > > > > https://github.com/apache/incubator-druid/issues/7866.
> > However
> > > > we
> > > > > > are
> > > > > > > > > still
> > > > > > > > > using Maven for now and may be for some time to come, since
> > it
> > > > > seems
> > > > > > > the
> > > > > > > > > benefits of switching to Gradle are definitely noticeable,
> > but
> > > > not
> > > > > > > huge.
> > > > > > > > >
> > > > > > > > > > - It doesn't appear that the Nexus artifacts require a
> GPG
> > > > > > signature
> > > > > > > > only
> > > > > > > > > > the DIST assembly requires GPG, which the script is
> > handling.
> > > > So
> > > > > > why
> > > > > > > > do
> > > > > > > > > we
> > > > > > > > > > need the gpg-plugin?
> > > > > > > > >
> > > > > > > > > IIRC it's because Sonatype's Central Repository (where the
> > > > > > Maven-ready
> > > > > > > > > artifacts ultimately end up) requires GPG signatures.
> > > > > > > > >
> > > > > > > > > > - It is not clear to me when a profile should be used
> > (e.g.,
> > > > for
> > > > > > > > deploy)
> > > > > > > > > vs
> > > > > > > > > > using the deploy process as part of the pom main build
> > > > lifecycle.
> > > > > > > > >
> > > > > > > > > I've found profiles to be most useful when you want
> something
> > > > like
> > > > > > "mvn
> > > > > > > > > package" to do different things based on a profile. e.g.
> > > having a
> > > > > > > profile
> > > > > > > > > to apply strict checks that takes extra time, having a
> > profile
> > > to
> > > > > > build
> > > > > > > > > extra assemblies, etc.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [Migration Help]

[Gian Merlino <gi...@apache.org>]

> What is confusing is that the Apache parent pom defines the
> maven-release-plugin
> only in build.pluginManagement where it calls the apache-release profile.
> Similarly, the Druid pom only defines the release plugin in
> build.pluginManagement.
> It has been my understanding that "pluginManagement" was only for
> inheritance.
> and that for a plugin to be actually executed it had to be referenced in
> the build.plugins
> section or in a profile.build.plugins.

This is getting outside of my Maven knowledge, but I would guess that the
release plugin is special and doesn't need to be listed as an explicit
plugin, since it has a special command to activate it (mvn release:prepare
or release:perform). It doesn't run on "normal" Maven commands.

> I don't see a profile called "dist".
> Where do I find that?

Both list and source-release-assembly-druid are in the "distribution" pom:
https://github.com/apache/incubator-druid/blob/master/distribution/pom.xml.
(Druid is a multi-module project, and the "distribution" module is the one
that builds the tarballs. The main pom 'sets the stage' but doesn't
actually do anything.)

On Fri, Aug 2, 2019 at 2:18 PM leerho <le...@gmail.com> wrote:

> >
> > It happens as part of the maven-release-plugin.
>
> What is confusing is that the Apache parent pom defines the
> maven-release-plugin
> only in build.pluginManagement where it calls the apache-release profile.
> Similarly, the Druid pom only defines the release plugin in
> build.pluginManagement.
> It has been my understanding that "pluginManagement" was only for
> inheritance.
> and that for a plugin to be actually executed it had to be referenced in
> the build.plugins
> section or in a profile.build.plugins.
>
>
>
> > - Check out the tag, then build release candidate with:
> > mvn clean install
> >     -Papache-release,dist,rat
> >     -DskipTests
> >     -Dgpg.keyname=<your GPG key fingerprint>
>
> I don't see a profile called "dist".
>
> We've defined our own, check out "source-release-assembly-druid".
>
>
> Where do I find that?
>
>
>
>
>
>
> On Fri, Aug 2, 2019 at 12:37 PM Gian Merlino <gi...@apache.org> wrote:
>
> > > 1) When you deploy the jar files to Nexus, I note that they are signed
> by
> > > GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> > > depending on the Apache parent pom "apache-release" profile, then where
> > is
> > > it being called?  From a script?
> >
> > It happens as part of the maven-release-plugin. The commands we run to
> do a
> > release candidate are something like:
> >
> > - Create a tag with: mvn -DreleaseVersion=0.14.1-incubating
> > -DdevelopmentVersion=0.14.2-incubating-SNAPSHOT
> > -Dtag=druid-0.14.1-incubating-rc1 -DpushChanges=false clean release:clean
> > release:prepare
> > - Check out the tag, then build release candidate with: mvn clean install
> > -Papache-release,dist,rat -DskipTests -Dgpg.keyname=<your GPG key
> > fingerprint>
> >
> > Later on, to upload the final artifacts to Nexus:
> >
> > - Using a saved release.properties file from (1), run: mvn
> release:perform
> > -Darguments=-Dgpg.keyname=<my-key>
> >
> > (If you lose the release.properties file, you should be able to recover
> > anyway by specifying a few more arguments on the release:perform line,
> but
> > I don't recall what they are off the top of my head.)
> >
> > > 2) You have a profile called "apache-release" that disables the
> > > source-release-assembly.  For the release to DIST, I presume you are
> > > building the tarball yourself, where?  The related question is what
> > script
> > > is using the above source-assembly.xml file to include the git.version
> +
> > > other stuff.?
> >
> > We've defined our own, check out "source-release-assembly-druid". We
> > disable the source-release-assembly inherited from the parent pom so we
> can
> > use ours instead. It gets built in the "mvn clean install" step above.
> >
> > > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > > documented.  Does this plugin introduce variables into the build
> process
> > > like ${git.commit.id}, ${git.build.version}, etc?  They are not
> defined
> > > elsewhere.  The jar-plugin needs the to insert into the manifest.
> >
> > I don't know the details, but that sounds like a very reasonable guess.
> >
> > On Fri, Aug 2, 2019 at 11:46 AM leerho <le...@gmail.com> wrote:
> >
> > > Two questions WRT the Druid root pom and deployment process
> > >
> > > 1) When you deploy the jar files to Nexus, I note that they are signed
> by
> > > GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> > > depending on the Apache parent pom "apache-release" profile, then where
> > is
> > > it being called?  From a script?
> > >
> > > 2) You have a profile called "apache-release" that disables the
> > > source-release-assembly.  For the release to DIST, I presume you are
> > > building the tarball yourself, where?  The related question is what
> > script
> > > is using the above source-assembly.xml file to include the git.version
> +
> > > other stuff.?
> > >
> > > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > > documented.  Does this plugin introduce variables into the build
> process
> > > like ${git.commit.id}, ${git.build.version}, etc?  They are not
> defined
> > > elsewhere.  The jar-plugin needs the to insert into the manifest.
> > >
> > >
> > >
> > > On Thu, Aug 1, 2019 at 11:42 AM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > > OK, I see it now in the tarball.  Where is the script or code that
> > > loads
> > > > > the data into git.version ?
> > > >
> > > > It's done by git-commit-id-plugin in our main POM:
> > > > https://github.com/apache/incubator-druid/blob/master/pom.xml. Then
> > it's
> > > > included into the source assembly via:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml
> > > >
> > > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just
> the
> > > PMC
> > > > > that votes on a release?
> > > >
> > > > Anyone can vote, but only PMC votes are binding. Even top level
> > projects
> > > > still should have a 72 hour voting period for each release candidate,
> > > > though, which can make release cycles take longer than you might be
> > used
> > > > to.
> > > >
> > > > On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:
> > > >
> > > > > >
> > > > > > We do have it in the tarball root for the real release
> > > > >
> > > > > OK, I see it now in the tarball.  Where is the script or code that
> > > loads
> > > > > the data into git.version ?
> > > > >
> > > > > If they stay separate...
> > > > >
> > > > >
> > > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just
> the
> > > PMC
> > > > > that votes on a release?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > > > > I gather you decided not to have a git.version file at the root
> > and
> > > > you
> > > > > > > added that info into the jar manifests, so where do you have
> that
> > > > > > > information for the tar.gz assemblies?
> > > > > >
> > > > > > We do have it in the tarball root for the real release -- the
> > source
> > > > > > tarball (ASF policy considers source releases as the sole
> official
> > > > > > release). For the binary convenience artifacts, we don't include
> it
> > > at
> > > > > the
> > > > > > root, but the jars have the info if people go digging.
> > > > > >
> > > > > > > Even the Java-only components have vastly different
> dependencies
> > > and
> > > > > > > version cycles and don't necessarily need to be released at the
> > > same
> > > > > time
> > > > > > > the core-java component is released.  So far, we don't have
> > > releases
> > > > > that
> > > > > > > span multiple components.  For example, the core-java library
> may
> > > be
> > > > > > > several versions ahead of the hive or pig components.
> > > > > >
> > > > > > Sure, that makes sense. I would still consider aligning their
> > release
> > > > > > processes though. If they stay separate, each component release
> > would
> > > > > need
> > > > > > its own release manager, its own voting cycle, its own publish
> and
> > > > > > announcement process, etc, and that can be a lot of
> administrative
> > > > > > overhead.
> > > > > >
> > > > > > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
> > > > > >
> > > > > > > Gian,
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > > > https://github.com/apache/incubator-druid/pull/6482.
> > > > > > >
> > > > > > >
> > > > > > > The discussion about the traceability of the GA back to the
> final
> > > RC
> > > > is
> > > > > > > interesting.
> > > > > > > I gather you decided not to have a git.version file at the root
> > and
> > > > you
> > > > > > > added that info into the jar manifests, so where do you have
> that
> > > > > > > information for the tar.gz assemblies?
> > > > > > >
> > > > > > > Also, it is more work upfront, but you might find that in the
> > long
> > > > run,
> > > > > > you
> > > > > > > > enjoy switching the project to a single repo with a
> > multi-module
> > > > > Maven
> > > > > > > > build. We do this in Druid and it works quite well. It makes
> it
> > > way
> > > > > > > easier
> > > > > > > > to do patches and releases that span multiple modules.
> > > > > > >
> > > > > > >
> > > > > > > The DataSketches library is not like a single product with
> > > components
> > > > > > that
> > > > > > > are released together.  Some of our "components" have different
> > > > > > life-cycles
> > > > > > > and versioning.  Also, we have C++ / Python repositories that
> > don't
> > > > fit
> > > > > > > well with Maven, which is all about Java.
> > > > > > >
> > > > > > > Even the Java-only components have vastly different
> dependencies
> > > and
> > > > > > > version cycles and don't necessarily need to be released at the
> > > same
> > > > > time
> > > > > > > the core-java component is released.  So far, we don't have
> > > releases
> > > > > that
> > > > > > > span multiple components.  For example, the core-java library
> may
> > > be
> > > > > > > several versions ahead of the hive or pig components.
> > > > > > >
> > > > > > > Definitely doing as much as possible in the build system
> (whether
> > > > Maven
> > > > > > or
> > > > > > > > Gradle) is a good thing, IMO, since they tend to reward
> living
> > in
> > > > > their
> > > > > > > > worlds. Also the more verifications you can push into the
> build
> > > > tool,
> > > > > > the
> > > > > > > > better.
> > > > > > >
> > > > > > >
> > > > > > > I agree.
> > > > > > >
> > > > > > > GPG ?
> > > > > > >
> > > > > > >
> > > > > > > You are correct.  The .asc signature is generated with GPG.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Hey Lee,
> > > > > > > >
> > > > > > > > You might find this PR useful, it is where we Apache-fied the
> > > Druid
> > > > > > POM:
> > > > > > > > https://github.com/apache/incubator-druid/pull/6482. It
> helps
> > > > > > highlight
> > > > > > > > the
> > > > > > > > specific areas that are relevant to Apache-fying a build.
> > > > > > > >
> > > > > > > > Also, it is more work upfront, but you might find that in the
> > > long
> > > > > run,
> > > > > > > you
> > > > > > > > enjoy switching the project to a single repo with a
> > multi-module
> > > > > Maven
> > > > > > > > build. We do this in Druid and it works quite well. It makes
> it
> > > way
> > > > > > > easier
> > > > > > > > to do patches and releases that span multiple modules.
> > > > > > > >
> > > > > > > > > So it seems that eventually, either we need to go whole hog
> > > > having
> > > > > > > Maven
> > > > > > > > do
> > > > > > > > > everything, which seems overly complicated, or consider
> > moving
> > > to
> > > > > > > Gradle
> > > > > > > > > (which I am not familiar with either, but it seems lots of
> > > teams
> > > > > are
> > > > > > > > moving
> > > > > > > > > that way).
> > > > > > > >
> > > > > > > > Definitely doing as much as possible in the build system
> > (whether
> > > > > Maven
> > > > > > > or
> > > > > > > > Gradle) is a good thing, IMO, since they tend to reward
> living
> > in
> > > > > their
> > > > > > > > worlds. Also the more verifications you can push into the
> build
> > > > tool,
> > > > > > the
> > > > > > > > better.
> > > > > > > >
> > > > > > > > In Druid we have also been considering moving to Gradle:
> > > > > > > > https://github.com/apache/incubator-druid/issues/7866.
> However
> > > we
> > > > > are
> > > > > > > > still
> > > > > > > > using Maven for now and may be for some time to come, since
> it
> > > > seems
> > > > > > the
> > > > > > > > benefits of switching to Gradle are definitely noticeable,
> but
> > > not
> > > > > > huge.
> > > > > > > >
> > > > > > > > > - It doesn't appear that the Nexus artifacts require a GPG
> > > > > signature
> > > > > > > only
> > > > > > > > > the DIST assembly requires GPG, which the script is
> handling.
> > > So
> > > > > why
> > > > > > > do
> > > > > > > > we
> > > > > > > > > need the gpg-plugin?
> > > > > > > >
> > > > > > > > IIRC it's because Sonatype's Central Repository (where the
> > > > > Maven-ready
> > > > > > > > artifacts ultimately end up) requires GPG signatures.
> > > > > > > >
> > > > > > > > > - It is not clear to me when a profile should be used
> (e.g.,
> > > for
> > > > > > > deploy)
> > > > > > > > vs
> > > > > > > > > using the deploy process as part of the pom main build
> > > lifecycle.
> > > > > > > >
> > > > > > > > I've found profiles to be most useful when you want something
> > > like
> > > > > "mvn
> > > > > > > > package" to do different things based on a profile. e.g.
> > having a
> > > > > > profile
> > > > > > > > to apply strict checks that takes extra time, having a
> profile
> > to
> > > > > build
> > > > > > > > extra assemblies, etc.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [Migration Help]

[leerho <le...@gmail.com>]

>
> It happens as part of the maven-release-plugin.

What is confusing is that the Apache parent pom defines the
maven-release-plugin
only in build.pluginManagement where it calls the apache-release profile.
Similarly, the Druid pom only defines the release plugin in
build.pluginManagement.
It has been my understanding that "pluginManagement" was only for
inheritance.
and that for a plugin to be actually executed it had to be referenced in
the build.plugins
section or in a profile.build.plugins.



> - Check out the tag, then build release candidate with:
> mvn clean install
>     -Papache-release,dist,rat
>     -DskipTests
>     -Dgpg.keyname=<your GPG key fingerprint>

I don't see a profile called "dist".

We've defined our own, check out "source-release-assembly-druid".


Where do I find that?






On Fri, Aug 2, 2019 at 12:37 PM Gian Merlino <gi...@apache.org> wrote:

> > 1) When you deploy the jar files to Nexus, I note that they are signed by
> > GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> > depending on the Apache parent pom "apache-release" profile, then where
> is
> > it being called?  From a script?
>
> It happens as part of the maven-release-plugin. The commands we run to do a
> release candidate are something like:
>
> - Create a tag with: mvn -DreleaseVersion=0.14.1-incubating
> -DdevelopmentVersion=0.14.2-incubating-SNAPSHOT
> -Dtag=druid-0.14.1-incubating-rc1 -DpushChanges=false clean release:clean
> release:prepare
> - Check out the tag, then build release candidate with: mvn clean install
> -Papache-release,dist,rat -DskipTests -Dgpg.keyname=<your GPG key
> fingerprint>
>
> Later on, to upload the final artifacts to Nexus:
>
> - Using a saved release.properties file from (1), run: mvn release:perform
> -Darguments=-Dgpg.keyname=<my-key>
>
> (If you lose the release.properties file, you should be able to recover
> anyway by specifying a few more arguments on the release:perform line, but
> I don't recall what they are off the top of my head.)
>
> > 2) You have a profile called "apache-release" that disables the
> > source-release-assembly.  For the release to DIST, I presume you are
> > building the tarball yourself, where?  The related question is what
> script
> > is using the above source-assembly.xml file to include the git.version +
> > other stuff.?
>
> We've defined our own, check out "source-release-assembly-druid". We
> disable the source-release-assembly inherited from the parent pom so we can
> use ours instead. It gets built in the "mvn clean install" step above.
>
> > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > documented.  Does this plugin introduce variables into the build process
> > like ${git.commit.id}, ${git.build.version}, etc?  They are not defined
> > elsewhere.  The jar-plugin needs the to insert into the manifest.
>
> I don't know the details, but that sounds like a very reasonable guess.
>
> On Fri, Aug 2, 2019 at 11:46 AM leerho <le...@gmail.com> wrote:
>
> > Two questions WRT the Druid root pom and deployment process
> >
> > 1) When you deploy the jar files to Nexus, I note that they are signed by
> > GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> > depending on the Apache parent pom "apache-release" profile, then where
> is
> > it being called?  From a script?
> >
> > 2) You have a profile called "apache-release" that disables the
> > source-release-assembly.  For the release to DIST, I presume you are
> > building the tarball yourself, where?  The related question is what
> script
> > is using the above source-assembly.xml file to include the git.version +
> > other stuff.?
> >
> > 3) I have now studied the "git-commit-id-plugin". Nice and well
> > documented.  Does this plugin introduce variables into the build process
> > like ${git.commit.id}, ${git.build.version}, etc?  They are not defined
> > elsewhere.  The jar-plugin needs the to insert into the manifest.
> >
> >
> >
> > On Thu, Aug 1, 2019 at 11:42 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > > OK, I see it now in the tarball.  Where is the script or code that
> > loads
> > > > the data into git.version ?
> > >
> > > It's done by git-commit-id-plugin in our main POM:
> > > https://github.com/apache/incubator-druid/blob/master/pom.xml. Then
> it's
> > > included into the source assembly via:
> > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml
> > >
> > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the
> > PMC
> > > > that votes on a release?
> > >
> > > Anyone can vote, but only PMC votes are binding. Even top level
> projects
> > > still should have a 72 hour voting period for each release candidate,
> > > though, which can make release cycles take longer than you might be
> used
> > > to.
> > >
> > > On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:
> > >
> > > > >
> > > > > We do have it in the tarball root for the real release
> > > >
> > > > OK, I see it now in the tarball.  Where is the script or code that
> > loads
> > > > the data into git.version ?
> > > >
> > > > If they stay separate...
> > > >
> > > >
> > > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the
> > PMC
> > > > that votes on a release?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > > I gather you decided not to have a git.version file at the root
> and
> > > you
> > > > > > added that info into the jar manifests, so where do you have that
> > > > > > information for the tar.gz assemblies?
> > > > >
> > > > > We do have it in the tarball root for the real release -- the
> source
> > > > > tarball (ASF policy considers source releases as the sole official
> > > > > release). For the binary convenience artifacts, we don't include it
> > at
> > > > the
> > > > > root, but the jars have the info if people go digging.
> > > > >
> > > > > > Even the Java-only components have vastly different dependencies
> > and
> > > > > > version cycles and don't necessarily need to be released at the
> > same
> > > > time
> > > > > > the core-java component is released.  So far, we don't have
> > releases
> > > > that
> > > > > > span multiple components.  For example, the core-java library may
> > be
> > > > > > several versions ahead of the hive or pig components.
> > > > >
> > > > > Sure, that makes sense. I would still consider aligning their
> release
> > > > > processes though. If they stay separate, each component release
> would
> > > > need
> > > > > its own release manager, its own voting cycle, its own publish and
> > > > > announcement process, etc, and that can be a lot of administrative
> > > > > overhead.
> > > > >
> > > > > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
> > > > >
> > > > > > Gian,
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > > > https://github.com/apache/incubator-druid/pull/6482.
> > > > > >
> > > > > >
> > > > > > The discussion about the traceability of the GA back to the final
> > RC
> > > is
> > > > > > interesting.
> > > > > > I gather you decided not to have a git.version file at the root
> and
> > > you
> > > > > > added that info into the jar manifests, so where do you have that
> > > > > > information for the tar.gz assemblies?
> > > > > >
> > > > > > Also, it is more work upfront, but you might find that in the
> long
> > > run,
> > > > > you
> > > > > > > enjoy switching the project to a single repo with a
> multi-module
> > > > Maven
> > > > > > > build. We do this in Druid and it works quite well. It makes it
> > way
> > > > > > easier
> > > > > > > to do patches and releases that span multiple modules.
> > > > > >
> > > > > >
> > > > > > The DataSketches library is not like a single product with
> > components
> > > > > that
> > > > > > are released together.  Some of our "components" have different
> > > > > life-cycles
> > > > > > and versioning.  Also, we have C++ / Python repositories that
> don't
> > > fit
> > > > > > well with Maven, which is all about Java.
> > > > > >
> > > > > > Even the Java-only components have vastly different dependencies
> > and
> > > > > > version cycles and don't necessarily need to be released at the
> > same
> > > > time
> > > > > > the core-java component is released.  So far, we don't have
> > releases
> > > > that
> > > > > > span multiple components.  For example, the core-java library may
> > be
> > > > > > several versions ahead of the hive or pig components.
> > > > > >
> > > > > > Definitely doing as much as possible in the build system (whether
> > > Maven
> > > > > or
> > > > > > > Gradle) is a good thing, IMO, since they tend to reward living
> in
> > > > their
> > > > > > > worlds. Also the more verifications you can push into the build
> > > tool,
> > > > > the
> > > > > > > better.
> > > > > >
> > > > > >
> > > > > > I agree.
> > > > > >
> > > > > > GPG ?
> > > > > >
> > > > > >
> > > > > > You are correct.  The .asc signature is generated with GPG.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hey Lee,
> > > > > > >
> > > > > > > You might find this PR useful, it is where we Apache-fied the
> > Druid
> > > > > POM:
> > > > > > > https://github.com/apache/incubator-druid/pull/6482. It helps
> > > > > highlight
> > > > > > > the
> > > > > > > specific areas that are relevant to Apache-fying a build.
> > > > > > >
> > > > > > > Also, it is more work upfront, but you might find that in the
> > long
> > > > run,
> > > > > > you
> > > > > > > enjoy switching the project to a single repo with a
> multi-module
> > > > Maven
> > > > > > > build. We do this in Druid and it works quite well. It makes it
> > way
> > > > > > easier
> > > > > > > to do patches and releases that span multiple modules.
> > > > > > >
> > > > > > > > So it seems that eventually, either we need to go whole hog
> > > having
> > > > > > Maven
> > > > > > > do
> > > > > > > > everything, which seems overly complicated, or consider
> moving
> > to
> > > > > > Gradle
> > > > > > > > (which I am not familiar with either, but it seems lots of
> > teams
> > > > are
> > > > > > > moving
> > > > > > > > that way).
> > > > > > >
> > > > > > > Definitely doing as much as possible in the build system
> (whether
> > > > Maven
> > > > > > or
> > > > > > > Gradle) is a good thing, IMO, since they tend to reward living
> in
> > > > their
> > > > > > > worlds. Also the more verifications you can push into the build
> > > tool,
> > > > > the
> > > > > > > better.
> > > > > > >
> > > > > > > In Druid we have also been considering moving to Gradle:
> > > > > > > https://github.com/apache/incubator-druid/issues/7866. However
> > we
> > > > are
> > > > > > > still
> > > > > > > using Maven for now and may be for some time to come, since it
> > > seems
> > > > > the
> > > > > > > benefits of switching to Gradle are definitely noticeable, but
> > not
> > > > > huge.
> > > > > > >
> > > > > > > > - It doesn't appear that the Nexus artifacts require a GPG
> > > > signature
> > > > > > only
> > > > > > > > the DIST assembly requires GPG, which the script is handling.
> > So
> > > > why
> > > > > > do
> > > > > > > we
> > > > > > > > need the gpg-plugin?
> > > > > > >
> > > > > > > IIRC it's because Sonatype's Central Repository (where the
> > > > Maven-ready
> > > > > > > artifacts ultimately end up) requires GPG signatures.
> > > > > > >
> > > > > > > > - It is not clear to me when a profile should be used (e.g.,
> > for
> > > > > > deploy)
> > > > > > > vs
> > > > > > > > using the deploy process as part of the pom main build
> > lifecycle.
> > > > > > >
> > > > > > > I've found profiles to be most useful when you want something
> > like
> > > > "mvn
> > > > > > > package" to do different things based on a profile. e.g.
> having a
> > > > > profile
> > > > > > > to apply strict checks that takes extra time, having a profile
> to
> > > > build
> > > > > > > extra assemblies, etc.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [Migration Help]

[Gian Merlino <gi...@apache.org>]

> 1) When you deploy the jar files to Nexus, I note that they are signed by
> GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> depending on the Apache parent pom "apache-release" profile, then where is
> it being called?  From a script?

It happens as part of the maven-release-plugin. The commands we run to do a
release candidate are something like:

- Create a tag with: mvn -DreleaseVersion=0.14.1-incubating
-DdevelopmentVersion=0.14.2-incubating-SNAPSHOT
-Dtag=druid-0.14.1-incubating-rc1 -DpushChanges=false clean release:clean
release:prepare
- Check out the tag, then build release candidate with: mvn clean install
-Papache-release,dist,rat -DskipTests -Dgpg.keyname=<your GPG key
fingerprint>

Later on, to upload the final artifacts to Nexus:

- Using a saved release.properties file from (1), run: mvn release:perform
-Darguments=-Dgpg.keyname=<my-key>

(If you lose the release.properties file, you should be able to recover
anyway by specifying a few more arguments on the release:perform line, but
I don't recall what they are off the top of my head.)

> 2) You have a profile called "apache-release" that disables the
> source-release-assembly.  For the release to DIST, I presume you are
> building the tarball yourself, where?  The related question is what script
> is using the above source-assembly.xml file to include the git.version +
> other stuff.?

We've defined our own, check out "source-release-assembly-druid". We
disable the source-release-assembly inherited from the parent pom so we can
use ours instead. It gets built in the "mvn clean install" step above.

> 3) I have now studied the "git-commit-id-plugin". Nice and well
> documented.  Does this plugin introduce variables into the build process
> like ${git.commit.id}, ${git.build.version}, etc?  They are not defined
> elsewhere.  The jar-plugin needs the to insert into the manifest.

I don't know the details, but that sounds like a very reasonable guess.

On Fri, Aug 2, 2019 at 11:46 AM leerho <le...@gmail.com> wrote:

> Two questions WRT the Druid root pom and deployment process
>
> 1) When you deploy the jar files to Nexus, I note that they are signed by
> GPG, yet, you do not have a GPG plugin defined in your pom. If you are
> depending on the Apache parent pom "apache-release" profile, then where is
> it being called?  From a script?
>
> 2) You have a profile called "apache-release" that disables the
> source-release-assembly.  For the release to DIST, I presume you are
> building the tarball yourself, where?  The related question is what script
> is using the above source-assembly.xml file to include the git.version +
> other stuff.?
>
> 3) I have now studied the "git-commit-id-plugin". Nice and well
> documented.  Does this plugin introduce variables into the build process
> like ${git.commit.id}, ${git.build.version}, etc?  They are not defined
> elsewhere.  The jar-plugin needs the to insert into the manifest.
>
>
>
> On Thu, Aug 1, 2019 at 11:42 AM Gian Merlino <gi...@apache.org> wrote:
>
> > > OK, I see it now in the tarball.  Where is the script or code that
> loads
> > > the data into git.version ?
> >
> > It's done by git-commit-id-plugin in our main POM:
> > https://github.com/apache/incubator-druid/blob/master/pom.xml. Then it's
> > included into the source assembly via:
> >
> >
> https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml
> >
> > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the
> PMC
> > > that votes on a release?
> >
> > Anyone can vote, but only PMC votes are binding. Even top level projects
> > still should have a 72 hour voting period for each release candidate,
> > though, which can make release cycles take longer than you might be used
> > to.
> >
> > On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:
> >
> > > >
> > > > We do have it in the tarball root for the real release
> > >
> > > OK, I see it now in the tarball.  Where is the script or code that
> loads
> > > the data into git.version ?
> > >
> > > If they stay separate...
> > >
> > >
> > > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the
> PMC
> > > that votes on a release?
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > > I gather you decided not to have a git.version file at the root and
> > you
> > > > > added that info into the jar manifests, so where do you have that
> > > > > information for the tar.gz assemblies?
> > > >
> > > > We do have it in the tarball root for the real release -- the source
> > > > tarball (ASF policy considers source releases as the sole official
> > > > release). For the binary convenience artifacts, we don't include it
> at
> > > the
> > > > root, but the jars have the info if people go digging.
> > > >
> > > > > Even the Java-only components have vastly different dependencies
> and
> > > > > version cycles and don't necessarily need to be released at the
> same
> > > time
> > > > > the core-java component is released.  So far, we don't have
> releases
> > > that
> > > > > span multiple components.  For example, the core-java library may
> be
> > > > > several versions ahead of the hive or pig components.
> > > >
> > > > Sure, that makes sense. I would still consider aligning their release
> > > > processes though. If they stay separate, each component release would
> > > need
> > > > its own release manager, its own voting cycle, its own publish and
> > > > announcement process, etc, and that can be a lot of administrative
> > > > overhead.
> > > >
> > > > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
> > > >
> > > > > Gian,
> > > > > Thanks!
> > > > >
> > > > >
> > > > > > https://github.com/apache/incubator-druid/pull/6482.
> > > > >
> > > > >
> > > > > The discussion about the traceability of the GA back to the final
> RC
> > is
> > > > > interesting.
> > > > > I gather you decided not to have a git.version file at the root and
> > you
> > > > > added that info into the jar manifests, so where do you have that
> > > > > information for the tar.gz assemblies?
> > > > >
> > > > > Also, it is more work upfront, but you might find that in the long
> > run,
> > > > you
> > > > > > enjoy switching the project to a single repo with a multi-module
> > > Maven
> > > > > > build. We do this in Druid and it works quite well. It makes it
> way
> > > > > easier
> > > > > > to do patches and releases that span multiple modules.
> > > > >
> > > > >
> > > > > The DataSketches library is not like a single product with
> components
> > > > that
> > > > > are released together.  Some of our "components" have different
> > > > life-cycles
> > > > > and versioning.  Also, we have C++ / Python repositories that don't
> > fit
> > > > > well with Maven, which is all about Java.
> > > > >
> > > > > Even the Java-only components have vastly different dependencies
> and
> > > > > version cycles and don't necessarily need to be released at the
> same
> > > time
> > > > > the core-java component is released.  So far, we don't have
> releases
> > > that
> > > > > span multiple components.  For example, the core-java library may
> be
> > > > > several versions ahead of the hive or pig components.
> > > > >
> > > > > Definitely doing as much as possible in the build system (whether
> > Maven
> > > > or
> > > > > > Gradle) is a good thing, IMO, since they tend to reward living in
> > > their
> > > > > > worlds. Also the more verifications you can push into the build
> > tool,
> > > > the
> > > > > > better.
> > > > >
> > > > >
> > > > > I agree.
> > > > >
> > > > > GPG ?
> > > > >
> > > > >
> > > > > You are correct.  The .asc signature is generated with GPG.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > > > Hey Lee,
> > > > > >
> > > > > > You might find this PR useful, it is where we Apache-fied the
> Druid
> > > > POM:
> > > > > > https://github.com/apache/incubator-druid/pull/6482. It helps
> > > > highlight
> > > > > > the
> > > > > > specific areas that are relevant to Apache-fying a build.
> > > > > >
> > > > > > Also, it is more work upfront, but you might find that in the
> long
> > > run,
> > > > > you
> > > > > > enjoy switching the project to a single repo with a multi-module
> > > Maven
> > > > > > build. We do this in Druid and it works quite well. It makes it
> way
> > > > > easier
> > > > > > to do patches and releases that span multiple modules.
> > > > > >
> > > > > > > So it seems that eventually, either we need to go whole hog
> > having
> > > > > Maven
> > > > > > do
> > > > > > > everything, which seems overly complicated, or consider moving
> to
> > > > > Gradle
> > > > > > > (which I am not familiar with either, but it seems lots of
> teams
> > > are
> > > > > > moving
> > > > > > > that way).
> > > > > >
> > > > > > Definitely doing as much as possible in the build system (whether
> > > Maven
> > > > > or
> > > > > > Gradle) is a good thing, IMO, since they tend to reward living in
> > > their
> > > > > > worlds. Also the more verifications you can push into the build
> > tool,
> > > > the
> > > > > > better.
> > > > > >
> > > > > > In Druid we have also been considering moving to Gradle:
> > > > > > https://github.com/apache/incubator-druid/issues/7866. However
> we
> > > are
> > > > > > still
> > > > > > using Maven for now and may be for some time to come, since it
> > seems
> > > > the
> > > > > > benefits of switching to Gradle are definitely noticeable, but
> not
> > > > huge.
> > > > > >
> > > > > > > - It doesn't appear that the Nexus artifacts require a GPG
> > > signature
> > > > > only
> > > > > > > the DIST assembly requires GPG, which the script is handling.
> So
> > > why
> > > > > do
> > > > > > we
> > > > > > > need the gpg-plugin?
> > > > > >
> > > > > > IIRC it's because Sonatype's Central Repository (where the
> > > Maven-ready
> > > > > > artifacts ultimately end up) requires GPG signatures.
> > > > > >
> > > > > > > - It is not clear to me when a profile should be used (e.g.,
> for
> > > > > deploy)
> > > > > > vs
> > > > > > > using the deploy process as part of the pom main build
> lifecycle.
> > > > > >
> > > > > > I've found profiles to be most useful when you want something
> like
> > > "mvn
> > > > > > package" to do different things based on a profile. e.g. having a
> > > > profile
> > > > > > to apply strict checks that takes extra time, having a profile to
> > > build
> > > > > > extra assemblies, etc.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [Migration Help]

[leerho <le...@gmail.com>]

Two questions WRT the Druid root pom and deployment process

1) When you deploy the jar files to Nexus, I note that they are signed by
GPG, yet, you do not have a GPG plugin defined in your pom. If you are
depending on the Apache parent pom "apache-release" profile, then where is
it being called?  From a script?

2) You have a profile called "apache-release" that disables the
source-release-assembly.  For the release to DIST, I presume you are
building the tarball yourself, where?  The related question is what script
is using the above source-assembly.xml file to include the git.version +
other stuff.?

3) I have now studied the "git-commit-id-plugin". Nice and well
documented.  Does this plugin introduce variables into the build process
like ${git.commit.id}, ${git.build.version}, etc?  They are not defined
elsewhere.  The jar-plugin needs the to insert into the manifest.



On Thu, Aug 1, 2019 at 11:42 AM Gian Merlino <gi...@apache.org> wrote:

> > OK, I see it now in the tarball.  Where is the script or code that loads
> > the data into git.version ?
>
> It's done by git-commit-id-plugin in our main POM:
> https://github.com/apache/incubator-druid/blob/master/pom.xml. Then it's
> included into the source assembly via:
>
> https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml
>
> > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the PMC
> > that votes on a release?
>
> Anyone can vote, but only PMC votes are binding. Even top level projects
> still should have a 72 hour voting period for each release candidate,
> though, which can make release cycles take longer than you might be used
> to.
>
> On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:
>
> > >
> > > We do have it in the tarball root for the real release
> >
> > OK, I see it now in the tarball.  Where is the script or code that loads
> > the data into git.version ?
> >
> > If they stay separate...
> >
> >
> > Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the PMC
> > that votes on a release?
> >
> >
> >
> >
> >
> > On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > > I gather you decided not to have a git.version file at the root and
> you
> > > > added that info into the jar manifests, so where do you have that
> > > > information for the tar.gz assemblies?
> > >
> > > We do have it in the tarball root for the real release -- the source
> > > tarball (ASF policy considers source releases as the sole official
> > > release). For the binary convenience artifacts, we don't include it at
> > the
> > > root, but the jars have the info if people go digging.
> > >
> > > > Even the Java-only components have vastly different dependencies and
> > > > version cycles and don't necessarily need to be released at the same
> > time
> > > > the core-java component is released.  So far, we don't have releases
> > that
> > > > span multiple components.  For example, the core-java library may be
> > > > several versions ahead of the hive or pig components.
> > >
> > > Sure, that makes sense. I would still consider aligning their release
> > > processes though. If they stay separate, each component release would
> > need
> > > its own release manager, its own voting cycle, its own publish and
> > > announcement process, etc, and that can be a lot of administrative
> > > overhead.
> > >
> > > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
> > >
> > > > Gian,
> > > > Thanks!
> > > >
> > > >
> > > > > https://github.com/apache/incubator-druid/pull/6482.
> > > >
> > > >
> > > > The discussion about the traceability of the GA back to the final RC
> is
> > > > interesting.
> > > > I gather you decided not to have a git.version file at the root and
> you
> > > > added that info into the jar manifests, so where do you have that
> > > > information for the tar.gz assemblies?
> > > >
> > > > Also, it is more work upfront, but you might find that in the long
> run,
> > > you
> > > > > enjoy switching the project to a single repo with a multi-module
> > Maven
> > > > > build. We do this in Druid and it works quite well. It makes it way
> > > > easier
> > > > > to do patches and releases that span multiple modules.
> > > >
> > > >
> > > > The DataSketches library is not like a single product with components
> > > that
> > > > are released together.  Some of our "components" have different
> > > life-cycles
> > > > and versioning.  Also, we have C++ / Python repositories that don't
> fit
> > > > well with Maven, which is all about Java.
> > > >
> > > > Even the Java-only components have vastly different dependencies and
> > > > version cycles and don't necessarily need to be released at the same
> > time
> > > > the core-java component is released.  So far, we don't have releases
> > that
> > > > span multiple components.  For example, the core-java library may be
> > > > several versions ahead of the hive or pig components.
> > > >
> > > > Definitely doing as much as possible in the build system (whether
> Maven
> > > or
> > > > > Gradle) is a good thing, IMO, since they tend to reward living in
> > their
> > > > > worlds. Also the more verifications you can push into the build
> tool,
> > > the
> > > > > better.
> > > >
> > > >
> > > > I agree.
> > > >
> > > > GPG ?
> > > >
> > > >
> > > > You are correct.  The .asc signature is generated with GPG.
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > Hey Lee,
> > > > >
> > > > > You might find this PR useful, it is where we Apache-fied the Druid
> > > POM:
> > > > > https://github.com/apache/incubator-druid/pull/6482. It helps
> > > highlight
> > > > > the
> > > > > specific areas that are relevant to Apache-fying a build.
> > > > >
> > > > > Also, it is more work upfront, but you might find that in the long
> > run,
> > > > you
> > > > > enjoy switching the project to a single repo with a multi-module
> > Maven
> > > > > build. We do this in Druid and it works quite well. It makes it way
> > > > easier
> > > > > to do patches and releases that span multiple modules.
> > > > >
> > > > > > So it seems that eventually, either we need to go whole hog
> having
> > > > Maven
> > > > > do
> > > > > > everything, which seems overly complicated, or consider moving to
> > > > Gradle
> > > > > > (which I am not familiar with either, but it seems lots of teams
> > are
> > > > > moving
> > > > > > that way).
> > > > >
> > > > > Definitely doing as much as possible in the build system (whether
> > Maven
> > > > or
> > > > > Gradle) is a good thing, IMO, since they tend to reward living in
> > their
> > > > > worlds. Also the more verifications you can push into the build
> tool,
> > > the
> > > > > better.
> > > > >
> > > > > In Druid we have also been considering moving to Gradle:
> > > > > https://github.com/apache/incubator-druid/issues/7866. However we
> > are
> > > > > still
> > > > > using Maven for now and may be for some time to come, since it
> seems
> > > the
> > > > > benefits of switching to Gradle are definitely noticeable, but not
> > > huge.
> > > > >
> > > > > > - It doesn't appear that the Nexus artifacts require a GPG
> > signature
> > > > only
> > > > > > the DIST assembly requires GPG, which the script is handling.  So
> > why
> > > > do
> > > > > we
> > > > > > need the gpg-plugin?
> > > > >
> > > > > IIRC it's because Sonatype's Central Repository (where the
> > Maven-ready
> > > > > artifacts ultimately end up) requires GPG signatures.
> > > > >
> > > > > > - It is not clear to me when a profile should be used (e.g., for
> > > > deploy)
> > > > > vs
> > > > > > using the deploy process as part of the pom main build lifecycle.
> > > > >
> > > > > I've found profiles to be most useful when you want something like
> > "mvn
> > > > > package" to do different things based on a profile. e.g. having a
> > > profile
> > > > > to apply strict checks that takes extra time, having a profile to
> > build
> > > > > extra assemblies, etc.
> > > > >
> > > >
> > >
> >
>

Re: [Migration Help]

[Gian Merlino <gi...@apache.org>]

> OK, I see it now in the tarball.  Where is the script or code that loads
> the data into git.version ?

It's done by git-commit-id-plugin in our main POM:
https://github.com/apache/incubator-druid/blob/master/pom.xml. Then it's
included into the source assembly via:
https://github.com/apache/incubator-druid/blob/master/distribution/src/assembly/source-assembly.xml

> Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the PMC
> that votes on a release?

Anyone can vote, but only PMC votes are binding. Even top level projects
still should have a 72 hour voting period for each release candidate,
though, which can make release cycles take longer than you might be used to.

On Thu, Aug 1, 2019 at 11:06 AM leerho <le...@gmail.com> wrote:

> >
> > We do have it in the tarball root for the real release
>
> OK, I see it now in the tarball.  Where is the script or code that loads
> the data into git.version ?
>
> If they stay separate...
>
>
> Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the PMC
> that votes on a release?
>
>
>
>
>
> On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org> wrote:
>
> > > I gather you decided not to have a git.version file at the root and you
> > > added that info into the jar manifests, so where do you have that
> > > information for the tar.gz assemblies?
> >
> > We do have it in the tarball root for the real release -- the source
> > tarball (ASF policy considers source releases as the sole official
> > release). For the binary convenience artifacts, we don't include it at
> the
> > root, but the jars have the info if people go digging.
> >
> > > Even the Java-only components have vastly different dependencies and
> > > version cycles and don't necessarily need to be released at the same
> time
> > > the core-java component is released.  So far, we don't have releases
> that
> > > span multiple components.  For example, the core-java library may be
> > > several versions ahead of the hive or pig components.
> >
> > Sure, that makes sense. I would still consider aligning their release
> > processes though. If they stay separate, each component release would
> need
> > its own release manager, its own voting cycle, its own publish and
> > announcement process, etc, and that can be a lot of administrative
> > overhead.
> >
> > On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
> >
> > > Gian,
> > > Thanks!
> > >
> > >
> > > > https://github.com/apache/incubator-druid/pull/6482.
> > >
> > >
> > > The discussion about the traceability of the GA back to the final RC is
> > > interesting.
> > > I gather you decided not to have a git.version file at the root and you
> > > added that info into the jar manifests, so where do you have that
> > > information for the tar.gz assemblies?
> > >
> > > Also, it is more work upfront, but you might find that in the long run,
> > you
> > > > enjoy switching the project to a single repo with a multi-module
> Maven
> > > > build. We do this in Druid and it works quite well. It makes it way
> > > easier
> > > > to do patches and releases that span multiple modules.
> > >
> > >
> > > The DataSketches library is not like a single product with components
> > that
> > > are released together.  Some of our "components" have different
> > life-cycles
> > > and versioning.  Also, we have C++ / Python repositories that don't fit
> > > well with Maven, which is all about Java.
> > >
> > > Even the Java-only components have vastly different dependencies and
> > > version cycles and don't necessarily need to be released at the same
> time
> > > the core-java component is released.  So far, we don't have releases
> that
> > > span multiple components.  For example, the core-java library may be
> > > several versions ahead of the hive or pig components.
> > >
> > > Definitely doing as much as possible in the build system (whether Maven
> > or
> > > > Gradle) is a good thing, IMO, since they tend to reward living in
> their
> > > > worlds. Also the more verifications you can push into the build tool,
> > the
> > > > better.
> > >
> > >
> > > I agree.
> > >
> > > GPG ?
> > >
> > >
> > > You are correct.  The .asc signature is generated with GPG.
> > >
> > >
> > >
> > >
> > > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > Hey Lee,
> > > >
> > > > You might find this PR useful, it is where we Apache-fied the Druid
> > POM:
> > > > https://github.com/apache/incubator-druid/pull/6482. It helps
> > highlight
> > > > the
> > > > specific areas that are relevant to Apache-fying a build.
> > > >
> > > > Also, it is more work upfront, but you might find that in the long
> run,
> > > you
> > > > enjoy switching the project to a single repo with a multi-module
> Maven
> > > > build. We do this in Druid and it works quite well. It makes it way
> > > easier
> > > > to do patches and releases that span multiple modules.
> > > >
> > > > > So it seems that eventually, either we need to go whole hog having
> > > Maven
> > > > do
> > > > > everything, which seems overly complicated, or consider moving to
> > > Gradle
> > > > > (which I am not familiar with either, but it seems lots of teams
> are
> > > > moving
> > > > > that way).
> > > >
> > > > Definitely doing as much as possible in the build system (whether
> Maven
> > > or
> > > > Gradle) is a good thing, IMO, since they tend to reward living in
> their
> > > > worlds. Also the more verifications you can push into the build tool,
> > the
> > > > better.
> > > >
> > > > In Druid we have also been considering moving to Gradle:
> > > > https://github.com/apache/incubator-druid/issues/7866. However we
> are
> > > > still
> > > > using Maven for now and may be for some time to come, since it seems
> > the
> > > > benefits of switching to Gradle are definitely noticeable, but not
> > huge.
> > > >
> > > > > - It doesn't appear that the Nexus artifacts require a GPG
> signature
> > > only
> > > > > the DIST assembly requires GPG, which the script is handling.  So
> why
> > > do
> > > > we
> > > > > need the gpg-plugin?
> > > >
> > > > IIRC it's because Sonatype's Central Repository (where the
> Maven-ready
> > > > artifacts ultimately end up) requires GPG signatures.
> > > >
> > > > > - It is not clear to me when a profile should be used (e.g., for
> > > deploy)
> > > > vs
> > > > > using the deploy process as part of the pom main build lifecycle.
> > > >
> > > > I've found profiles to be most useful when you want something like
> "mvn
> > > > package" to do different things based on a profile. e.g. having a
> > profile
> > > > to apply strict checks that takes extra time, having a profile to
> build
> > > > extra assemblies, etc.
> > > >
> > >
> >
>

Re: [Migration Help]

[leerho <le...@gmail.com>]

>
> We do have it in the tarball root for the real release

OK, I see it now in the tarball.  Where is the script or code that loads
the data into git.version ?

If they stay separate...


Yes, I understand.  Once we graduate (PPMC -> PMC), isn't it just the PMC
that votes on a release?





On Wed, Jul 31, 2019 at 7:46 PM Gian Merlino <gi...@apache.org> wrote:

> > I gather you decided not to have a git.version file at the root and you
> > added that info into the jar manifests, so where do you have that
> > information for the tar.gz assemblies?
>
> We do have it in the tarball root for the real release -- the source
> tarball (ASF policy considers source releases as the sole official
> release). For the binary convenience artifacts, we don't include it at the
> root, but the jars have the info if people go digging.
>
> > Even the Java-only components have vastly different dependencies and
> > version cycles and don't necessarily need to be released at the same time
> > the core-java component is released.  So far, we don't have releases that
> > span multiple components.  For example, the core-java library may be
> > several versions ahead of the hive or pig components.
>
> Sure, that makes sense. I would still consider aligning their release
> processes though. If they stay separate, each component release would need
> its own release manager, its own voting cycle, its own publish and
> announcement process, etc, and that can be a lot of administrative
> overhead.
>
> On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:
>
> > Gian,
> > Thanks!
> >
> >
> > > https://github.com/apache/incubator-druid/pull/6482.
> >
> >
> > The discussion about the traceability of the GA back to the final RC is
> > interesting.
> > I gather you decided not to have a git.version file at the root and you
> > added that info into the jar manifests, so where do you have that
> > information for the tar.gz assemblies?
> >
> > Also, it is more work upfront, but you might find that in the long run,
> you
> > > enjoy switching the project to a single repo with a multi-module Maven
> > > build. We do this in Druid and it works quite well. It makes it way
> > easier
> > > to do patches and releases that span multiple modules.
> >
> >
> > The DataSketches library is not like a single product with components
> that
> > are released together.  Some of our "components" have different
> life-cycles
> > and versioning.  Also, we have C++ / Python repositories that don't fit
> > well with Maven, which is all about Java.
> >
> > Even the Java-only components have vastly different dependencies and
> > version cycles and don't necessarily need to be released at the same time
> > the core-java component is released.  So far, we don't have releases that
> > span multiple components.  For example, the core-java library may be
> > several versions ahead of the hive or pig components.
> >
> > Definitely doing as much as possible in the build system (whether Maven
> or
> > > Gradle) is a good thing, IMO, since they tend to reward living in their
> > > worlds. Also the more verifications you can push into the build tool,
> the
> > > better.
> >
> >
> > I agree.
> >
> > GPG ?
> >
> >
> > You are correct.  The .asc signature is generated with GPG.
> >
> >
> >
> >
> > On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > Hey Lee,
> > >
> > > You might find this PR useful, it is where we Apache-fied the Druid
> POM:
> > > https://github.com/apache/incubator-druid/pull/6482. It helps
> highlight
> > > the
> > > specific areas that are relevant to Apache-fying a build.
> > >
> > > Also, it is more work upfront, but you might find that in the long run,
> > you
> > > enjoy switching the project to a single repo with a multi-module Maven
> > > build. We do this in Druid and it works quite well. It makes it way
> > easier
> > > to do patches and releases that span multiple modules.
> > >
> > > > So it seems that eventually, either we need to go whole hog having
> > Maven
> > > do
> > > > everything, which seems overly complicated, or consider moving to
> > Gradle
> > > > (which I am not familiar with either, but it seems lots of teams are
> > > moving
> > > > that way).
> > >
> > > Definitely doing as much as possible in the build system (whether Maven
> > or
> > > Gradle) is a good thing, IMO, since they tend to reward living in their
> > > worlds. Also the more verifications you can push into the build tool,
> the
> > > better.
> > >
> > > In Druid we have also been considering moving to Gradle:
> > > https://github.com/apache/incubator-druid/issues/7866. However we are
> > > still
> > > using Maven for now and may be for some time to come, since it seems
> the
> > > benefits of switching to Gradle are definitely noticeable, but not
> huge.
> > >
> > > > - It doesn't appear that the Nexus artifacts require a GPG signature
> > only
> > > > the DIST assembly requires GPG, which the script is handling.  So why
> > do
> > > we
> > > > need the gpg-plugin?
> > >
> > > IIRC it's because Sonatype's Central Repository (where the Maven-ready
> > > artifacts ultimately end up) requires GPG signatures.
> > >
> > > > - It is not clear to me when a profile should be used (e.g., for
> > deploy)
> > > vs
> > > > using the deploy process as part of the pom main build lifecycle.
> > >
> > > I've found profiles to be most useful when you want something like "mvn
> > > package" to do different things based on a profile. e.g. having a
> profile
> > > to apply strict checks that takes extra time, having a profile to build
> > > extra assemblies, etc.
> > >
> >
>

Re: [Migration Help]

[Gian Merlino <gi...@apache.org>]

> I gather you decided not to have a git.version file at the root and you
> added that info into the jar manifests, so where do you have that
> information for the tar.gz assemblies?

We do have it in the tarball root for the real release -- the source
tarball (ASF policy considers source releases as the sole official
release). For the binary convenience artifacts, we don't include it at the
root, but the jars have the info if people go digging.

> Even the Java-only components have vastly different dependencies and
> version cycles and don't necessarily need to be released at the same time
> the core-java component is released.  So far, we don't have releases that
> span multiple components.  For example, the core-java library may be
> several versions ahead of the hive or pig components.

Sure, that makes sense. I would still consider aligning their release
processes though. If they stay separate, each component release would need
its own release manager, its own voting cycle, its own publish and
announcement process, etc, and that can be a lot of administrative overhead.

On Wed, Jul 31, 2019 at 7:07 PM leerho <le...@gmail.com> wrote:

> Gian,
> Thanks!
>
>
> > https://github.com/apache/incubator-druid/pull/6482.
>
>
> The discussion about the traceability of the GA back to the final RC is
> interesting.
> I gather you decided not to have a git.version file at the root and you
> added that info into the jar manifests, so where do you have that
> information for the tar.gz assemblies?
>
> Also, it is more work upfront, but you might find that in the long run, you
> > enjoy switching the project to a single repo with a multi-module Maven
> > build. We do this in Druid and it works quite well. It makes it way
> easier
> > to do patches and releases that span multiple modules.
>
>
> The DataSketches library is not like a single product with components that
> are released together.  Some of our "components" have different life-cycles
> and versioning.  Also, we have C++ / Python repositories that don't fit
> well with Maven, which is all about Java.
>
> Even the Java-only components have vastly different dependencies and
> version cycles and don't necessarily need to be released at the same time
> the core-java component is released.  So far, we don't have releases that
> span multiple components.  For example, the core-java library may be
> several versions ahead of the hive or pig components.
>
> Definitely doing as much as possible in the build system (whether Maven or
> > Gradle) is a good thing, IMO, since they tend to reward living in their
> > worlds. Also the more verifications you can push into the build tool, the
> > better.
>
>
> I agree.
>
> GPG ?
>
>
> You are correct.  The .asc signature is generated with GPG.
>
>
>
>
> On Tue, Jul 30, 2019 at 5:22 PM Gian Merlino <gi...@apache.org> wrote:
>
> > Hey Lee,
> >
> > You might find this PR useful, it is where we Apache-fied the Druid POM:
> > https://github.com/apache/incubator-druid/pull/6482. It helps highlight
> > the
> > specific areas that are relevant to Apache-fying a build.
> >
> > Also, it is more work upfront, but you might find that in the long run,
> you
> > enjoy switching the project to a single repo with a multi-module Maven
> > build. We do this in Druid and it works quite well. It makes it way
> easier
> > to do patches and releases that span multiple modules.
> >
> > > So it seems that eventually, either we need to go whole hog having
> Maven
> > do
> > > everything, which seems overly complicated, or consider moving to
> Gradle
> > > (which I am not familiar with either, but it seems lots of teams are
> > moving
> > > that way).
> >
> > Definitely doing as much as possible in the build system (whether Maven
> or
> > Gradle) is a good thing, IMO, since they tend to reward living in their
> > worlds. Also the more verifications you can push into the build tool, the
> > better.
> >
> > In Druid we have also been considering moving to Gradle:
> > https://github.com/apache/incubator-druid/issues/7866. However we are
> > still
> > using Maven for now and may be for some time to come, since it seems the
> > benefits of switching to Gradle are definitely noticeable, but not huge.
> >
> > > - It doesn't appear that the Nexus artifacts require a GPG signature
> only
> > > the DIST assembly requires GPG, which the script is handling.  So why
> do
> > we
> > > need the gpg-plugin?
> >
> > IIRC it's because Sonatype's Central Repository (where the Maven-ready
> > artifacts ultimately end up) requires GPG signatures.
> >
> > > - It is not clear to me when a profile should be used (e.g., for
> deploy)
> > vs
> > > using the deploy process as part of the pom main build lifecycle.
> >
> > I've found profiles to be most useful when you want something like "mvn
> > package" to do different things based on a profile. e.g. having a profile
> > to apply strict checks that takes extra time, having a profile to build
> > extra assemblies, etc.
> >
>