Re: [DISCUSSION] Apache Ignite Extension Release Process Improvements

[Maxim Muzafarov <mm...@apache.org>]

Folks,


1. The changes mentioned above have been implemented in the following
umbrella ticket [1]. Please, take a look.
2. Removing dependency from the `log4j-test.xml` or `tests.properties`
in extensions should be a part of [2] issue implementation (removing
dependency from log4j 1.2.17).


== CHANGES ==


1. The ignite-parent pom artifact is now added to the Ignite release
lifecycle. It will allow sharing the checkstyle, properties, profiles
and build configurations to the Ignite Extension which in turn leads
to the single building + releasing lifecycle for the Ignite + Ignite
Extensions.

2. The BOM module created with all the valuable Ignite modules. It
simplifies the internal dependency management. You don't need use
<version>${project.verstion}<version> for adding a new module as a
dependency.

3. All changes are automatically deployed to the Maven Central
Snapshot repository on each commit in the master branch (GitHub Action
is configured for this). This will simplify the extensions development
lifecycle on the top of the Apache Ignite changes.
https://github.com/apache/ignite/blob/master/.github/workflows/publish-snapshot.yml
https://repository.apache.org/content/repositories/snapshots/org/apache/ignite/ignite-core/2.14.0-SNAPSHOT/


== RELEASE EXAMPLE ==


1.
The release branch created for the Zooleeper Ip Finder Extension
https://github.com/apache/ignite-extensions/blob/release/ignite-zookeeper-ip-finder-ext-1.0.0/modules/zookeeper-ip-finder-ext/pom.xml#L34

2.
See the GitHub Action with prepared RC. The Action must be stared from
the release branch (release/ignite-zookeeper-ip-finder-ext-1.0.0). You
can download the prepared `artifact` and sign the release with your
own signature.
https://github.com/apache/ignite-extensions/actions/runs/2239023047

During the release candidate preparation:

- validate the release branch
(release/ignite-zookeeper-ip-findere-ext-1.0.0 must have 1.0.0 version
in the pom);
- checking that there are no SNAPSHOT versions in the release branch;
- build the sources and binaries (if applicable) archives for the
releasing extension;
- preparing shell scripts for signing and deploying artifact to the
Maven Central staging and svn;
- creating a new rc-tag which points to the last commit in the release branch;

3.
See the prepared binaries and sources which were created and uploaded
using shell scripts from the Action artifact.
https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-zookeeper-ip-finder-ext-1.0.0-rc1/

4.
You can check and validate the release candidate using the following
GitHub Action fro the master branch:
https://github.com/apache/ignite-extensions/runs/6210193576?check_suite_focus=true

This action will do:

- compare version of the release branch and pom;
- build the extension sources;
- validate archives (binary and sources) hashes;
- validate  archives (binary and sources)  signatures;
- check that there is a rc-tag which points to the last commit;

5.
You can find the valuable release preparation instruction in the DEVNOTES
https://github.com/apache/ignite-extensions/blob/master/DEVNOTES.md

6.
I deliberately didn't add the post-release shell scripts since they
are really straightforward.
(close Maven staging from the UI, move binaries & sources to new
directory, add release tag)



[1] https://issues.apache.org/jira/browse/IGNITE-16855
[2] https://issues.apache.org/jira/browse/IGNITE-16650

On Fri, 18 Feb 2022 at 10:05, Nikolay Izhikov <ni...@apache.org> wrote:
>
> Hello, Maxim.
>
> Thanks for investigation!
>
> > 4. log4j-test.xml, tests.properties these files from the ignite-core module required to run tests and it seems they should be shared between projects also as tests resources.
>
> As you may see most copies of these files are empty.
> I think we should modify `GridAbstractTest` and other classes to use defaults if `log4j-test.xml` or `tests.properties` unavailable.
>
> > 4. Configure Travis [10] to deploy a new snapshot version to the apache maven snapshots repository [5] each time the Ignite master branch is updated.
>
> Can we attach Ignite sources as maven modules to Ignite extensions?
> And use Ignite sources as dependencies?
>
> > 18 февр. 2022 г., в 00:30, Maxim Muzafarov <mm...@apache.org> написал(а):
> >
> > Hello Igniters,
> >
> >
> > We have a lot of Ignite extensions waiting to be released (e.g. aws,
> > gce, topology-validator, ignite-spring-tx-ext etc.). I think it is a
> > good point in time to make some kind of automation and simplify our
> > release process for Ignite extension modules removing manual steps
> > from it.
> >
> >
> > During the investigation I've found some issues that we have:
> >
> > 1. Ignite extensions have their own parent pom file [2]. The code and
> > dependency versions are fully duplicated with the Ignite parent pom
> > [1], however, these properties must be shared for Ignite Extension
> > project. Take a look at the <jetty.version>:
> > - master Apache Ignite: 9.4.39.v20210325
> > - master Apache Ignite Extensions: 9.4.11.v20180605
> >
> > 2. The checkstyle configuration between these two repositories are
> > different [3] [4]. Each of Apache Ignite and Apache Ignite Extensions
> > projects have their own checkstyle config file, however, it must be
> > shared the same as the maven checkstyle profile.
> >
> > 3. The Extensions have a dependency on the latest Apache Ignite
> > version in the master branch (e.g. 2.13-SNAPSHOT), but without
> > deploying it in the local repository the Extension projects can be
> > built. It seems the latest version should be published to the maven
> > snapshots repository [5].
> >
> > 4. log4j-test.xml, tests.properties these files from the ignite-core
> > module required to run tests and it seems they should be shared
> > between projects also as tests resources.
> >
> > 5. The apache-release profile [7] from the apache parent which will
> > prepare the sources, checksums, gpg sign is not used in the release
> > process at all.
> >
> >
> > Here are my suggestions to fix all the issues mentioned above:
> >
> > 1. Create the `ignite-parent` module that will be released each time
> > with the latest release version. This module will be based on the
> > parent/pom.xml [1]. This will allow us to share all common
> > configurations, profiles, dependency versions and properties to the
> > Ignite Extensions.
> >
> > 2. Create the `ignite-plugin-bom` bill of materials [8] module with
> > the common Ignite dependencies required for developing a new
> > extension. This is a common practice for companies that have their own
> > parent pom file and can't inherit from Ignite's one (e.g.
> > spring-boot-bom example [9] with all spring dependencies).
> >
> > 3. Create the additional `ignite-resources` jar file to share
> > checkstyle configuration, resources, tests resources and etc. for the
> > Extension project.
> >
> > 4. Configure Travis [10] to deploy a new snapshot version to the
> > apache maven snapshots repository [5] each time the Ignite master
> > branch is updated.
> >
> > 5. Clean up the pom.xml Extensions file from profiles and dependencies
> > that are not required to release extensions e.g. the release profile
> > [11].
> >
> > 6. Reuse the `apache-release` profile for preparing sources of the
> > extensions during the release.
> >
> >
> > WDYT?
> >
> >
> > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L35
> > [2] https://github.com/apache/ignite-extensions/blob/master/parent/pom.xml#L36
> > [3] https://github.com/apache/ignite-extensions/blob/master/checkstyle/checkstyle.xml
> > [4] https://github.com/apache/ignite/blob/master/checkstyle/checkstyle.xml
> > [5] https://repository.apache.org/content/repositories/snapshots
> > [6] https://github.com/apache/ignite-extensions/tree/master/modules/aws-ext/modules/core/src/test/config
> > [7] https://github.com/apache/maven-apache-parent/blob/apache-24/pom.xml#L375
> > [8] https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#bill-of-materials-bom-poms
> > [9] https://github.com/snowdrop/spring-boot-bom/blob/sb-2.5.x/pom.xml#L31
> > [10] https://blog.travis-ci.com/2017-03-30-deploy-maven-travis-ci-packagecloud/
> > [11] https://github.com/apache/ignite-extensions/blob/master/pom.xml#L242
>