Re: Separating distribution and maven artifacts in Nexus staging repo?

[Marshall Schor <ms...@schor.com>]

I wrote up this question on the Maven users list, asking about a Maven Best
Practice in this area, and got one interesting response.

First, some background:
The "convention over configuration" normal way for multi-module assemblies would
seem to be:

1) have an "assembly" project
2) When it is released, the Apache common release POM, using the conventional
"apache-release" profile, would run a pipeline that

built the assembly

got it GPG signed

got it uploaded (to the Nexus staging Repo).

I was initially not "attaching" the assembly in the Maven sense, to prevent it
from being uploaded (it's 16+ MB, and designed to be distributed using the
Apache mirroring system).  This did keep it from getting uploaded to Maven, but
also prevented the GPG plugin from signing it (That plugin only signs "attached"
artifacts), and also no sha/md5 checksums were generated.

Looking at the normal path in "apache-release" - I see that the deploy plugin
which uploads to the Nexus staging Repo is, internally, generating the md5/sha
checksums.

Here's the solution described on the Maven user's list for this:

1) keep the assembly "attached" - this is the default
2) prevent the "deploy" step from running - there's an override configuration
you can add called <skip>true</skip>

Then add the following to the apache-release profile activity:

1) override the install plugin - to cause it to add checksums (md5/sha) to the
.m2 repository.  There's a configuration for this in the install plugin.
2) add an "ant" execution step to copy from the .m2 repo (after the checksums
have been done), to the assembly target in a new directory, "assemblies".
3) add an execution of "wagon" plugin that uploads from this new directory to
where you want to have it "staged" to - for people to test it, etc.  This is not
the Nexus repo, since these assemblies aren't going into Nexus.

I can see how this would work.  I wonder what other projects are doing here, for
their multi-module source and binary assemblies?

Looking forward to hearing from a lot of projects ( I hope :-) ).

-Marshall Schor

On 7/27/2010 3:51 AM, ant elder wrote:
> I'd asked this on infrastructure@ but now found this list so i'll ask here too:
>
> Is there an easy way to separate out the distribution artifacts and
> maven artifacts when staging a release to Nexus so that when the
> release is promoted to the maven central repo the distribution
> artifacts don't also end up in there?
>
> Looking at a few other projects they don't seem to do anything special
> so everything goes into the maven central repo, so maybe it doesn't
> matter. Is this something i need to worry about? I'm asking related to
> Tuscany and the distributions can be quite large, eg bin tar and zip
> each about 70 Meg  -
> https://repository.apache.org/content/repositories/orgapachetuscany-027/org/apache/tuscany/sca/tuscany-distribution-all/2.0-Beta1/
>
> I'm looking for a release process with a single command does all the
> signing and uploading of all the artifacts so i can kick it off and go
> away while it happens.
>
>   ...ant
>
>

Re: Separating distribution and maven artifacts in Nexus staging repo?

[Marshall Schor <ms...@schor.com>]

On 11/5/2010 3:00 PM, Jukka Zitting wrote:
> Hi,
>
> On Fri, Nov 5, 2010 at 7:18 PM, Marshall Schor <ms...@schor.com> wrote:
>> Let the Nexus uploads to the staging repository occur for these artifacts; this
>> will add the checksums. [...] This seems to require the least special maven
>> POM code.
> Another alternative that we use in Jackrabbit, Tika and PDFBox is to
> simply disable deployment (using the skip option of the deploy plugin)
> of the top level reactor project that produces the source release
> package. This way the package never hits the staging repository, but
> we still get the GPG signatures with no extra configuration.

The other thing needed, when you do eventually push these artifacts to the
apache-distr spot, is to have the checksums created.  This is normally done
using the deploy plugin, so if you disable this using "skip", what do you use to
add the checksums?

-Marshall
> BR,
>
> Jukka Zitting
>
>

Re: Separating distribution and maven artifacts in Nexus staging repo?

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Nov 5, 2010 at 7:18 PM, Marshall Schor <ms...@schor.com> wrote:
> Let the Nexus uploads to the staging repository occur for these artifacts; this
> will add the checksums. [...] This seems to require the least special maven
> POM code.

Another alternative that we use in Jackrabbit, Tika and PDFBox is to
simply disable deployment (using the skip option of the deploy plugin)
of the top level reactor project that produces the source release
package. This way the package never hits the staging repository, but
we still get the GPG signatures with no extra configuration.

BR,

Jukka Zitting

Re: Separating distribution and maven artifacts in Nexus staging repo?

[sebb <se...@gmail.com>]

On 5 November 2010 18:57, Brian Fox <br...@infinity.nu> wrote:
> On Fri, Nov 5, 2010 at 2:50 PM, sebb <se...@gmail.com> wrote:
>> On 5 November 2010 18:45, Brian Fox <br...@infinity.nu> wrote:
>>>> If Nexus could be taught not to publish such items, then that would be great.
>>>
>>> We could probably do something like that as a rule on promotion but
>>> where should we put them?
>>>
>>
>> Why not just drop the files?
>>
> I'm thinking about the people who forgot to move them to dist first.
> If we want it to be a requirement, then I can just set the promote
> rules to forbid certain files. Then it can pass the first set of rules
> to be closed, with the files, but won't let them be promoted beyond
> that.

Yes, that sounds even better, especially if there is a clear error
message that is associated with the rule failure.

Re: Separating distribution and maven artifacts in Nexus staging repo?

[Brian Fox <br...@infinity.nu>]

On Fri, Nov 5, 2010 at 2:50 PM, sebb <se...@gmail.com> wrote:
> On 5 November 2010 18:45, Brian Fox <br...@infinity.nu> wrote:
>>> If Nexus could be taught not to publish such items, then that would be great.
>>
>> We could probably do something like that as a rule on promotion but
>> where should we put them?
>>
>
> Why not just drop the files?
>
I'm thinking about the people who forgot to move them to dist first.
If we want it to be a requirement, then I can just set the promote
rules to forbid certain files. Then it can pass the first set of rules
to be closed, with the files, but won't let them be promoted beyond
that.

Re: Separating distribution and maven artifacts in Nexus staging repo?

[sebb <se...@gmail.com>]

On 5 November 2010 18:45, Brian Fox <br...@infinity.nu> wrote:
>> If Nexus could be taught not to publish such items, then that would be great.
>
> We could probably do something like that as a rule on promotion but
> where should we put them?
>

Why not just drop the files?

Re: Separating distribution and maven artifacts in Nexus staging repo?

[Brian Fox <br...@infinity.nu>]

> If Nexus could be taught not to publish such items, then that would be great.

We could probably do something like that as a rule on promotion but
where should we put them?

Re: Separating distribution and maven artifacts in Nexus staging repo?

[sebb <se...@gmail.com>]

On 5 November 2010 17:18, Marshall Schor <ms...@schor.com> wrote:
> Here's my latest thoughts on this.
>
> Have the release process produce the assemblies as "attached" artifacts (so that
> the maven-gpg-plugin will do the signing).
>
> Let the Nexus uploads to the staging repository occur for these artifacts; this
> will add the checksums.  Although the assemblies are large, they are in a spot
> where other developers on the project can download them (with or without maven)
> and test the release.  If not put there, they would need to take up space
> somewhere else in the Apache infrastructure for sharing with the development
> community for testing, anyways.
>
> After the release vote, use Nexus GUI to delete these assemblies, before pushing
> the "release" button.  This prevents them from going up to Maven Central.  And,
> of course, copy the assemblies you want to have mirrored by the normal Apache
> distribution mechanism, to the spot for your project.
>
> This seems to require the least special maven POM code.  Does anyone see a flaw
> in this approach?

We did this in Commons recently (Exec I think).

The problem is that it is all too easy to accidentally publish the
non-Maven archives to Maven.

If Nexus could be taught not to publish such items, then that would be great.

> -Marshall Schor
>
> On 11/4/2010 5:45 PM, Marshall Schor wrote:
>> I wrote up this question on the Maven users list, asking about a Maven Best
>> Practice in this area, and got one interesting response.
>>
>> First, some background:
>> The "convention over configuration" normal way for multi-module assemblies would
>> seem to be:
>>
>> 1) have an "assembly" project
>> 2) When it is released, the Apache common release POM, using the conventional
>> "apache-release" profile, would run a pipeline that
>>
>> built the assembly
>>
>> got it GPG signed
>>
>> got it uploaded (to the Nexus staging Repo).
>>
>> I was initially not "attaching" the assembly in the Maven sense, to prevent it
>> from being uploaded (it's 16+ MB, and designed to be distributed using the
>> Apache mirroring system).  This did keep it from getting uploaded to Maven, but
>> also prevented the GPG plugin from signing it (That plugin only signs "attached"
>> artifacts), and also no sha/md5 checksums were generated.
>>
>> Looking at the normal path in "apache-release" - I see that the deploy plugin
>> which uploads to the Nexus staging Repo is, internally, generating the md5/sha
>> checksums.
>>
>> Here's the solution described on the Maven user's list for this:
>>
>> 1) keep the assembly "attached" - this is the default
>> 2) prevent the "deploy" step from running - there's an override configuration
>> you can add called <skip>true</skip>
>>
>> Then add the following to the apache-release profile activity:
>>
>> 1) override the install plugin - to cause it to add checksums (md5/sha) to the
>> .m2 repository.  There's a configuration for this in the install plugin.
>> 2) add an "ant" execution step to copy from the .m2 repo (after the checksums
>> have been done), to the assembly target in a new directory, "assemblies".
>> 3) add an execution of "wagon" plugin that uploads from this new directory to
>> where you want to have it "staged" to - for people to test it, etc.  This is not
>> the Nexus repo, since these assemblies aren't going into Nexus.
>>
>> I can see how this would work.  I wonder what other projects are doing here, for
>> their multi-module source and binary assemblies?
>>
>> Looking forward to hearing from a lot of projects ( I hope :-) ).
>>
>> -Marshall Schor
>>
>> On 7/27/2010 3:51 AM, ant elder wrote:
>>> I'd asked this on infrastructure@ but now found this list so i'll ask here too:
>>>
>>> Is there an easy way to separate out the distribution artifacts and
>>> maven artifacts when staging a release to Nexus so that when the
>>> release is promoted to the maven central repo the distribution
>>> artifacts don't also end up in there?
>>>
>>> Looking at a few other projects they don't seem to do anything special
>>> so everything goes into the maven central repo, so maybe it doesn't
>>> matter. Is this something i need to worry about? I'm asking related to
>>> Tuscany and the distributions can be quite large, eg bin tar and zip
>>> each about 70 Meg  -
>>> https://repository.apache.org/content/repositories/orgapachetuscany-027/org/apache/tuscany/sca/tuscany-distribution-all/2.0-Beta1/
>>>
>>> I'm looking for a release process with a single command does all the
>>> signing and uploading of all the artifacts so i can kick it off and go
>>> away while it happens.
>>>
>>>   ...ant
>>>
>>>
>>
>

Re: Separating distribution and maven artifacts in Nexus staging repo?

[Marshall Schor <ms...@schor.com>]

Here's my latest thoughts on this.

Have the release process produce the assemblies as "attached" artifacts (so that
the maven-gpg-plugin will do the signing).

Let the Nexus uploads to the staging repository occur for these artifacts; this
will add the checksums.  Although the assemblies are large, they are in a spot
where other developers on the project can download them (with or without maven)
and test the release.  If not put there, they would need to take up space
somewhere else in the Apache infrastructure for sharing with the development
community for testing, anyways.

After the release vote, use Nexus GUI to delete these assemblies, before pushing
the "release" button.  This prevents them from going up to Maven Central.  And,
of course, copy the assemblies you want to have mirrored by the normal Apache
distribution mechanism, to the spot for your project.

This seems to require the least special maven POM code.  Does anyone see a flaw
in this approach?

-Marshall Schor

On 11/4/2010 5:45 PM, Marshall Schor wrote:
> I wrote up this question on the Maven users list, asking about a Maven Best
> Practice in this area, and got one interesting response.
>
> First, some background:
> The "convention over configuration" normal way for multi-module assemblies would
> seem to be:
>
> 1) have an "assembly" project
> 2) When it is released, the Apache common release POM, using the conventional
> "apache-release" profile, would run a pipeline that
>
> built the assembly
>
> got it GPG signed
>
> got it uploaded (to the Nexus staging Repo).
>
> I was initially not "attaching" the assembly in the Maven sense, to prevent it
> from being uploaded (it's 16+ MB, and designed to be distributed using the
> Apache mirroring system).  This did keep it from getting uploaded to Maven, but
> also prevented the GPG plugin from signing it (That plugin only signs "attached"
> artifacts), and also no sha/md5 checksums were generated.
>
> Looking at the normal path in "apache-release" - I see that the deploy plugin
> which uploads to the Nexus staging Repo is, internally, generating the md5/sha
> checksums.
>
> Here's the solution described on the Maven user's list for this:
>
> 1) keep the assembly "attached" - this is the default
> 2) prevent the "deploy" step from running - there's an override configuration
> you can add called <skip>true</skip>
>
> Then add the following to the apache-release profile activity:
>
> 1) override the install plugin - to cause it to add checksums (md5/sha) to the
> .m2 repository.  There's a configuration for this in the install plugin.
> 2) add an "ant" execution step to copy from the .m2 repo (after the checksums
> have been done), to the assembly target in a new directory, "assemblies".
> 3) add an execution of "wagon" plugin that uploads from this new directory to
> where you want to have it "staged" to - for people to test it, etc.  This is not
> the Nexus repo, since these assemblies aren't going into Nexus.
>
> I can see how this would work.  I wonder what other projects are doing here, for
> their multi-module source and binary assemblies?
>
> Looking forward to hearing from a lot of projects ( I hope :-) ).
>
> -Marshall Schor
>
> On 7/27/2010 3:51 AM, ant elder wrote:
>> I'd asked this on infrastructure@ but now found this list so i'll ask here too:
>>
>> Is there an easy way to separate out the distribution artifacts and
>> maven artifacts when staging a release to Nexus so that when the
>> release is promoted to the maven central repo the distribution
>> artifacts don't also end up in there?
>>
>> Looking at a few other projects they don't seem to do anything special
>> so everything goes into the maven central repo, so maybe it doesn't
>> matter. Is this something i need to worry about? I'm asking related to
>> Tuscany and the distributions can be quite large, eg bin tar and zip
>> each about 70 Meg  -
>> https://repository.apache.org/content/repositories/orgapachetuscany-027/org/apache/tuscany/sca/tuscany-distribution-all/2.0-Beta1/
>>
>> I'm looking for a release process with a single command does all the
>> signing and uploading of all the artifacts so i can kick it off and go
>> away while it happens.
>>
>>   ...ant
>>
>>
>