You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Daniel Klco <dk...@apache.org> on 2019/06/13 15:20:11 UTC
Release Validator Docker Image
I've also been working on a docker image to validate and build releases:
https://github.com/apache/sling-whiteboard/tree/master/release-validator
It's relatively easy to use. You'll need to build it, then checking a
release is as easy as:
docker run sling-check-release [build-num]
A couple things I have noticed / want to change are:
- Support mounting a local folder to save the downloaded artifacts
- It doesn't support the situation where one release depends on another
independent release. It will, however, support dependent releases in the
same repository. I'm not sure there is a clean way around this.
Re: Release Validator Docker Image
Posted by Robert Munteanu <ro...@apache.org>.
Filed https://issues.apache.org/jira/browse/SLING-8699 for tracking.
Thanks,
Robert
On Wed, 2019-09-11 at 04:19 +0200, Carsten Ziegeler wrote:
> Sounds good to me, with the addition that it does not upload the md5
> files to SVN and only creates the sha256 (or sha512?) if the check
> of
> the files is successful
>
> Carsten
>
> Am 10.09.2019 um 23:33 schrieb Robert Munteanu:
> > On Tue, 2019-09-10 at 22:57 +0200, Carsten Ziegeler wrote:
> > > I think the value is already there today, we push manually to
> > > dist
> > > and
> > > having the sha512 there and not the md5 is already a win.
> > > It would be great to have this mirrored in the maven repository,
> > > but
> > > that probably needs those bugs to be fixed.
> >
> > So my proposal, echoing discussions at the hackathon would be to
> > implement a command that publishes a release to dist and that:
> >
> > - downloads the release artifacts from Nexus (already covered by
> > existing command)
> > - computes SHA-256 checksums and adds them to the files to be
> > deployed
> > - uses the Java API equivalent of `svn remove
> > https://dist.apache.org/
> > ...` to remove all old artifacts of a release
> > - uses the Java API equivalent of `svn import .
> > https://dist.apache.org/ ...` to import local set of artifacts to
> > SVN
> >
> > Would that cover everyone's needs for pushing to dist?
> >
> > Robert
> >
> > > Carsten
> > >
> > > Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
> > > > Hi Carsten,
> > > >
> > > > > On 9 Sep 2019, at 19:29, Carsten Ziegeler <
> > > > > cziegeler@apache.org>
> > > > > wrote:
> > > > >
> > > > > Can we maybe also remove the md5 files and create sha512
> > > > > files,
> > > > > when the checksum is correct?
> > > > >
> > > >
> > > > The sha1 and md5 files seem to be generated by Nexus and not by
> > > > our
> > > > Maven release process. I quickly checked how the sha512 files
> > > > are
> > > > generated and this seems to be a config in the Apache parent
> > > > pom
> > > > [4]. However, those are not attached [5] to the build, most
> > > > probably due to [6].
> > > >
> > > > I guess we can create the sha512 files with this CLI tool, but
> > > > the
> > > > value would come only when we’d be able to push the staged
> > > > artifacts to dist.
> > > >
> > > > Cheers,
> > > > Radu
> > > >
> > > > [4] -
> > > > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > > > <
> > > > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > > > [5] -
> > > > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums
> > > > <
> > > > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6
> > > > ] - https://issues.apache.org/jira/browse/INFRA-14923 <
> > > > https://issues.apache.org/jira/browse/INFRA-14923>
> > > >
Re: Release Validator Docker Image
Posted by Carsten Ziegeler <cz...@apache.org>.
Sounds good to me, with the addition that it does not upload the md5
files to SVN and only creates the sha256 (or sha512?) if the check of
the files is successful
Carsten
Am 10.09.2019 um 23:33 schrieb Robert Munteanu:
> On Tue, 2019-09-10 at 22:57 +0200, Carsten Ziegeler wrote:
>> I think the value is already there today, we push manually to dist
>> and
>> having the sha512 there and not the md5 is already a win.
>> It would be great to have this mirrored in the maven repository, but
>> that probably needs those bugs to be fixed.
>
> So my proposal, echoing discussions at the hackathon would be to
> implement a command that publishes a release to dist and that:
>
> - downloads the release artifacts from Nexus (already covered by
> existing command)
> - computes SHA-256 checksums and adds them to the files to be deployed
> - uses the Java API equivalent of `svn remove https://dist.apache.org/
> ...` to remove all old artifacts of a release
> - uses the Java API equivalent of `svn import .
> https://dist.apache.org/ ...` to import local set of artifacts to SVN
>
> Would that cover everyone's needs for pushing to dist?
>
> Robert
>
>>
>> Carsten
>>
>> Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
>>> Hi Carsten,
>>>
>>>> On 9 Sep 2019, at 19:29, Carsten Ziegeler <cz...@apache.org>
>>>> wrote:
>>>>
>>>> Can we maybe also remove the md5 files and create sha512 files,
>>>> when the checksum is correct?
>>>>
>>>
>>> The sha1 and md5 files seem to be generated by Nexus and not by our
>>> Maven release process. I quickly checked how the sha512 files are
>>> generated and this seems to be a config in the Apache parent pom
>>> [4]. However, those are not attached [5] to the build, most
>>> probably due to [6].
>>>
>>> I guess we can create the sha512 files with this CLI tool, but the
>>> value would come only when we’d be able to push the staged
>>> artifacts to dist.
>>>
>>> Cheers,
>>> Radu
>>>
>>> [4] -
>>> https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
>>> <
>>> https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
>>>>
>>> [5] -
>>> https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums
>>> <
>>> https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6
>>> ] - https://issues.apache.org/jira/browse/INFRA-14923 <
>>> https://issues.apache.org/jira/browse/INFRA-14923>
>>>
>
--
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Release Validator Docker Image
Posted by Robert Munteanu <ro...@apache.org>.
On Tue, 2019-09-10 at 22:57 +0200, Carsten Ziegeler wrote:
> I think the value is already there today, we push manually to dist
> and
> having the sha512 there and not the md5 is already a win.
> It would be great to have this mirrored in the maven repository, but
> that probably needs those bugs to be fixed.
So my proposal, echoing discussions at the hackathon would be to
implement a command that publishes a release to dist and that:
- downloads the release artifacts from Nexus (already covered by
existing command)
- computes SHA-256 checksums and adds them to the files to be deployed
- uses the Java API equivalent of `svn remove https://dist.apache.org/
...` to remove all old artifacts of a release
- uses the Java API equivalent of `svn import .
https://dist.apache.org/ ...` to import local set of artifacts to SVN
Would that cover everyone's needs for pushing to dist?
Robert
>
> Carsten
>
> Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
> > Hi Carsten,
> >
> > > On 9 Sep 2019, at 19:29, Carsten Ziegeler <cz...@apache.org>
> > > wrote:
> > >
> > > Can we maybe also remove the md5 files and create sha512 files,
> > > when the checksum is correct?
> > >
> >
> > The sha1 and md5 files seem to be generated by Nexus and not by our
> > Maven release process. I quickly checked how the sha512 files are
> > generated and this seems to be a config in the Apache parent pom
> > [4]. However, those are not attached [5] to the build, most
> > probably due to [6].
> >
> > I guess we can create the sha512 files with this CLI tool, but the
> > value would come only when we’d be able to push the staged
> > artifacts to dist.
> >
> > Cheers,
> > Radu
> >
> > [4] -
> > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > <
> > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > >
> > [5] -
> > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums
> > <
> > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6
> > ] - https://issues.apache.org/jira/browse/INFRA-14923 <
> > https://issues.apache.org/jira/browse/INFRA-14923>
> >
Re: Release Validator Docker Image
Posted by Carsten Ziegeler <cz...@apache.org>.
I think the value is already there today, we push manually to dist and
having the sha512 there and not the md5 is already a win.
It would be great to have this mirrored in the maven repository, but
that probably needs those bugs to be fixed.
Carsten
Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
> Hi Carsten,
>
>> On 9 Sep 2019, at 19:29, Carsten Ziegeler <cz...@apache.org> wrote:
>>
>> Can we maybe also remove the md5 files and create sha512 files, when the checksum is correct?
>>
>
> The sha1 and md5 files seem to be generated by Nexus and not by our Maven release process. I quickly checked how the sha512 files are generated and this seems to be a config in the Apache parent pom [4]. However, those are not attached [5] to the build, most probably due to [6].
>
> I guess we can create the sha512 files with this CLI tool, but the value would come only when we’d be able to push the staged artifacts to dist.
>
> Cheers,
> Radu
>
> [4] - https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440 <https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440>
> [5] - https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums <https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6] - https://issues.apache.org/jira/browse/INFRA-14923 <https://issues.apache.org/jira/browse/INFRA-14923>
>
--
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Release Validator Docker Image
Posted by Radu Cotescu <ra...@apache.org>.
Hi Carsten,
> On 9 Sep 2019, at 19:29, Carsten Ziegeler <cz...@apache.org> wrote:
>
> Can we maybe also remove the md5 files and create sha512 files, when the checksum is correct?
>
The sha1 and md5 files seem to be generated by Nexus and not by our Maven release process. I quickly checked how the sha512 files are generated and this seems to be a config in the Apache parent pom [4]. However, those are not attached [5] to the build, most probably due to [6].
I guess we can create the sha512 files with this CLI tool, but the value would come only when we’d be able to push the staged artifacts to dist.
Cheers,
Radu
[4] - https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440 <https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440>
[5] - https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums <https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6] - https://issues.apache.org/jira/browse/INFRA-14923 <https://issues.apache.org/jira/browse/INFRA-14923>
Re: Release Validator Docker Image
Posted by Carsten Ziegeler <cz...@apache.org>.
Can we maybe also remove the md5 files and create sha512 files, when the
checksum is correct?
Carsten
Am 09.09.2019 um 19:25 schrieb Daniel Klco:
> Thanks Radu for doing the hard work. As predicted, once that was in place,
> the CI check was relatively straightforward. I added the code to check the
> CI status from GitHub, updated the GPG check to download the Sling key file
> and added a summary message letting the user know if all of the checks
> succeeded:
>
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/310b49d9d57ba97cc651468ab14143f15433b202
>
> You should now be able to do the following:
>
> bash-3.2$ docker run --env-file=./docker-env apache/sling-cli release
> verify --repository=2126
>
>
> org.apache.sling.fileoptim-0.9.4-source-release.zip
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (ea25bb7fb0cb0bff51f20eb4d21cb91d5262259d)
>
> MD-5: VALID (69c30bb2f5e117d1f4ba2d4278b15c0e)
>
>
> org.apache.sling.fileoptim-0.9.4.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (5a5df233209b018420f4255a3a087aeff8bb0a01)
>
> MD-5: VALID (8be14fc95cb74e1d3ff23869dfe9afef)
>
>
> org.apache.sling.fileoptim-0.9.4.pom
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (e08d7a115313af4c10e7006022d730e07a1cfc42)
>
> MD-5: VALID (cfc07c2782b95a3c41e0b8b61fff54f2)
>
> CI Status: VALID:
>
> continuous-integration/jenkins/branch
>
> State: success
>
> Description: This commit looks good
>
> See:
> https://builds.apache.org/job/Sling/job/sling-org-apache-sling-file-optimization/job/master/37/display/redirect
>
>
> org.apache.sling.fileoptim-0.9.4-javadoc.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (003ebb552c39c7df07613ed21e7d11d8cc51d27a)
>
> MD-5: VALID (f9b54cca8995a115b978716bb5ac9c14)
>
>
> org.apache.sling.fileoptim-0.9.4-sources.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (ff94c1c349390cd6e9e4316c648ce6909757bef2)
>
> MD-5: VALID (746e62e2e70c006ad083ac76c18da798)
>
>
>
> Release Summary: VALID (16 checks executed)
>
> On Mon, Sep 9, 2019 at 9:16 AM Daniel Klco <da...@gmail.com> wrote:
>
>> Nice! The approach using the Lucene search of the repo contents is much
>> cleaner than what I was thinking.
>>
>> On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <ra...@apache.org> wrote:
>>
>>> Hi Daniel,
>>>
>>>> On 6 Sep 2019, at 18:01, Daniel Klco <da...@gmail.com> wrote:
>>>>
>>>> Thanks Radu,
>>>>
>>>> I've been looking into this on and off over the last few days and I'm
>>> still
>>>> struggling to see the value of re-implementing all of this in Java
>>> instead
>>>> of relying on a simple series of scripts. Sure it's possible to do GPG
>>> key
>>>> and Hash validation, but I'd still need to write a scraper to download
>>> the
>>>> artifacts from Nexus and then all of the code around these other
>>> functions,
>>>> each of which are hundreds of lines of Java code vs a single line in
>>> BASH.
>>>>
>>>> What would you think about flipping this the other way around? Instead
>>> of
>>>> re-implementing in Java, make the docker image more flexible and use
>>>> scripts to decide which function to invoke?
>>>>
>>>> I've put together a quick example of what I'm thinking here:
>>>>
>>>>
>>> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>>>>
>>>
>>> I’ve managed to implement the missing pieces during last week’s hackathon
>>> and on Friday afternoon [3]. The fact that we ship a Docker image with that
>>> code is just convenience I guess. The functionality is implemented actually
>>> as an OSGi app using the Sling feature model, so we could pack it however
>>> we want.
>>>
>>> Cheers,
>>> Radu
>>>
>>> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
>>> https://issues.apache.org/jira/browse/SLING-8684>
>>
>>
>
--
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Thanks Radu for doing the hard work. As predicted, once that was in place,
the CI check was relatively straightforward. I added the code to check the
CI status from GitHub, updated the GPG check to download the Sling key file
and added a summary message letting the user know if all of the checks
succeeded:
https://github.com/apache/sling-org-apache-sling-committer-cli/commit/310b49d9d57ba97cc651468ab14143f15433b202
You should now be able to do the following:
bash-3.2$ docker run --env-file=./docker-env apache/sling-cli release
verify --repository=2126
org.apache.sling.fileoptim-0.9.4-source-release.zip
GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
SHA-1: VALID (ea25bb7fb0cb0bff51f20eb4d21cb91d5262259d)
MD-5: VALID (69c30bb2f5e117d1f4ba2d4278b15c0e)
org.apache.sling.fileoptim-0.9.4.jar
GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
SHA-1: VALID (5a5df233209b018420f4255a3a087aeff8bb0a01)
MD-5: VALID (8be14fc95cb74e1d3ff23869dfe9afef)
org.apache.sling.fileoptim-0.9.4.pom
GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
SHA-1: VALID (e08d7a115313af4c10e7006022d730e07a1cfc42)
MD-5: VALID (cfc07c2782b95a3c41e0b8b61fff54f2)
CI Status: VALID:
continuous-integration/jenkins/branch
State: success
Description: This commit looks good
See:
https://builds.apache.org/job/Sling/job/sling-org-apache-sling-file-optimization/job/master/37/display/redirect
org.apache.sling.fileoptim-0.9.4-javadoc.jar
GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
SHA-1: VALID (003ebb552c39c7df07613ed21e7d11d8cc51d27a)
MD-5: VALID (f9b54cca8995a115b978716bb5ac9c14)
org.apache.sling.fileoptim-0.9.4-sources.jar
GPG: signed by Dan Klco (CODE SIGNING KEY) <dk...@apache.org> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
SHA-1: VALID (ff94c1c349390cd6e9e4316c648ce6909757bef2)
MD-5: VALID (746e62e2e70c006ad083ac76c18da798)
Release Summary: VALID (16 checks executed)
On Mon, Sep 9, 2019 at 9:16 AM Daniel Klco <da...@gmail.com> wrote:
> Nice! The approach using the Lucene search of the repo contents is much
> cleaner than what I was thinking.
>
> On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <ra...@apache.org> wrote:
>
>> Hi Daniel,
>>
>> > On 6 Sep 2019, at 18:01, Daniel Klco <da...@gmail.com> wrote:
>> >
>> > Thanks Radu,
>> >
>> > I've been looking into this on and off over the last few days and I'm
>> still
>> > struggling to see the value of re-implementing all of this in Java
>> instead
>> > of relying on a simple series of scripts. Sure it's possible to do GPG
>> key
>> > and Hash validation, but I'd still need to write a scraper to download
>> the
>> > artifacts from Nexus and then all of the code around these other
>> functions,
>> > each of which are hundreds of lines of Java code vs a single line in
>> BASH.
>> >
>> > What would you think about flipping this the other way around? Instead
>> of
>> > re-implementing in Java, make the docker image more flexible and use
>> > scripts to decide which function to invoke?
>> >
>> > I've put together a quick example of what I'm thinking here:
>> >
>> >
>> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>> >
>>
>> I’ve managed to implement the missing pieces during last week’s hackathon
>> and on Friday afternoon [3]. The fact that we ship a Docker image with that
>> code is just convenience I guess. The functionality is implemented actually
>> as an OSGi app using the Sling feature model, so we could pack it however
>> we want.
>>
>> Cheers,
>> Radu
>>
>> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
>> https://issues.apache.org/jira/browse/SLING-8684>
>
>
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Nice! The approach using the Lucene search of the repo contents is much
cleaner than what I was thinking.
On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <ra...@apache.org> wrote:
> Hi Daniel,
>
> > On 6 Sep 2019, at 18:01, Daniel Klco <da...@gmail.com> wrote:
> >
> > Thanks Radu,
> >
> > I've been looking into this on and off over the last few days and I'm
> still
> > struggling to see the value of re-implementing all of this in Java
> instead
> > of relying on a simple series of scripts. Sure it's possible to do GPG
> key
> > and Hash validation, but I'd still need to write a scraper to download
> the
> > artifacts from Nexus and then all of the code around these other
> functions,
> > each of which are hundreds of lines of Java code vs a single line in
> BASH.
> >
> > What would you think about flipping this the other way around? Instead of
> > re-implementing in Java, make the docker image more flexible and use
> > scripts to decide which function to invoke?
> >
> > I've put together a quick example of what I'm thinking here:
> >
> >
> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
> >
>
> I’ve managed to implement the missing pieces during last week’s hackathon
> and on Friday afternoon [3]. The fact that we ship a Docker image with that
> code is just convenience I guess. The functionality is implemented actually
> as an OSGi app using the Sling feature model, so we could pack it however
> we want.
>
> Cheers,
> Radu
>
> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
> https://issues.apache.org/jira/browse/SLING-8684>
Re: Release Validator Docker Image
Posted by Radu Cotescu <ra...@apache.org>.
Hi Daniel,
> On 6 Sep 2019, at 18:01, Daniel Klco <da...@gmail.com> wrote:
>
> Thanks Radu,
>
> I've been looking into this on and off over the last few days and I'm still
> struggling to see the value of re-implementing all of this in Java instead
> of relying on a simple series of scripts. Sure it's possible to do GPG key
> and Hash validation, but I'd still need to write a scraper to download the
> artifacts from Nexus and then all of the code around these other functions,
> each of which are hundreds of lines of Java code vs a single line in BASH.
>
> What would you think about flipping this the other way around? Instead of
> re-implementing in Java, make the docker image more flexible and use
> scripts to decide which function to invoke?
>
> I've put together a quick example of what I'm thinking here:
>
> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>
I’ve managed to implement the missing pieces during last week’s hackathon and on Friday afternoon [3]. The fact that we ship a Docker image with that code is just convenience I guess. The functionality is implemented actually as an OSGi app using the Sling feature model, so we could pack it however we want.
Cheers,
Radu
[3] - https://issues.apache.org/jira/browse/SLING-8684 <https://issues.apache.org/jira/browse/SLING-8684>
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Thanks Radu,
I've been looking into this on and off over the last few days and I'm still
struggling to see the value of re-implementing all of this in Java instead
of relying on a simple series of scripts. Sure it's possible to do GPG key
and Hash validation, but I'd still need to write a scraper to download the
artifacts from Nexus and then all of the code around these other functions,
each of which are hundreds of lines of Java code vs a single line in BASH.
What would you think about flipping this the other way around? Instead of
re-implementing in Java, make the docker image more flexible and use
scripts to decide which function to invoke?
I've put together a quick example of what I'm thinking here:
https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
On Tue, Aug 27, 2019 at 3:31 AM Radu Cotescu <ra...@apache.org> wrote:
> Hi Daniel,
>
> > On 23 Aug 2019, at 14:59, Daniel Klco <da...@gmail.com> wrote:
> >
> > I'm just not convinced of the
> > value re-implementing in Java brings vs a few simple bash commands.
>
> Sure, it’s more work to do, but you’d benefit from a nice command line
> menu and we would unify the tooling. Here are a few examples:
>
> $ docker run -it --env-file=./docker-env apache/sling-cli
> Usage: docker run -it --env-file=./docker-env apache/sling-cli [COMMAND]
> Apache Sling Committers CLI
> Commands:
> help Displays help information about the specified command
> release Performs release-related operations
>
>
>
> $ docker run -it --env-file=./docker-env apache/sling-cli release help
> Usage: docker run -it --env-file=./docker-env apache/sling-cli release
> [COMMAND]
> Performs release-related operations
> Commands:
> create-new-jira-version Creates a new Jira version, if needed, and
> transitions any unresolved issues from the version being released to
> the next one
> list Lists all open releases
> prepare-email Prepares an email vote for the releases found
> in the Nexus staged repository
> tally-votes Counts votes cast for a release and generates
> the result email
> update-local-site Updates the Sling website with the new release
> information, based on a local checkout
> update-reporter Updates the Apache Reporter System with the new
> release information
> help Displays help information about the specified
> command
>
>
>
> $ docker run -it --env-file=./docker-env apache/sling-cli release
> prepare-email help
> Usage: docker run -it --env-file=./docker-env apache/sling-cli release
> prepare-email -r=<repositoryId> [-x=<executionMode>] [COMMAND]
> Prepares an email vote for the releases found in the Nexus staged
> repository
> -r, --repository=<repositoryId>
> Nexus repository id
> -x, --execution-mode=<executionMode>
> execution mode: DRY_RUN, INTERACTIVE, AUTO; default: DRY_RUN
> Commands:
> help Displays help information about the specified command
>
>
>
> To help with this I’ve already pushed some code for verifying release
> signatures in a branch [2], which could easily be integrated with the rest.
> Checking build status is trivial by comparison.
>
> Regards,
> Radu
>
> [2] -
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6
> <
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6
> >
Re: Release Validator Docker Image
Posted by Radu Cotescu <ra...@apache.org>.
Hi Daniel,
> On 23 Aug 2019, at 14:59, Daniel Klco <da...@gmail.com> wrote:
>
> I'm just not convinced of the
> value re-implementing in Java brings vs a few simple bash commands.
Sure, it’s more work to do, but you’d benefit from a nice command line menu and we would unify the tooling. Here are a few examples:
$ docker run -it --env-file=./docker-env apache/sling-cli
Usage: docker run -it --env-file=./docker-env apache/sling-cli [COMMAND]
Apache Sling Committers CLI
Commands:
help Displays help information about the specified command
release Performs release-related operations
$ docker run -it --env-file=./docker-env apache/sling-cli release help
Usage: docker run -it --env-file=./docker-env apache/sling-cli release [COMMAND]
Performs release-related operations
Commands:
create-new-jira-version Creates a new Jira version, if needed, and transitions any unresolved issues from the version being released to
the next one
list Lists all open releases
prepare-email Prepares an email vote for the releases found in the Nexus staged repository
tally-votes Counts votes cast for a release and generates the result email
update-local-site Updates the Sling website with the new release information, based on a local checkout
update-reporter Updates the Apache Reporter System with the new release information
help Displays help information about the specified command
$ docker run -it --env-file=./docker-env apache/sling-cli release prepare-email help
Usage: docker run -it --env-file=./docker-env apache/sling-cli release prepare-email -r=<repositoryId> [-x=<executionMode>] [COMMAND]
Prepares an email vote for the releases found in the Nexus staged repository
-r, --repository=<repositoryId>
Nexus repository id
-x, --execution-mode=<executionMode>
execution mode: DRY_RUN, INTERACTIVE, AUTO; default: DRY_RUN
Commands:
help Displays help information about the specified command
To help with this I’ve already pushed some code for verifying release signatures in a branch [2], which could easily be integrated with the rest. Checking build status is trivial by comparison.
Regards,
Radu
[2] - https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6 <https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6>
Re: Release Validator Docker Image
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Fri, Aug 23, 2019 at 3:00 PM Daniel Klco <da...@gmail.com> wrote:
> ...I'm just not convinced of the
> value re-implementing in Java brings vs a few simple bash commands...
+1, IMO what's important is for the tools used to be traceable, for
example by including digests of scripts (or Docker images?) in their
output and having people validate those digests before running the
tools.
The OpenWhisk release checking script [1] does that:
echo "$(basename $0) (script SHA1: $(gpg --print-md SHA1 $0 | cut
-d' ' -f2-))"
So that when someone pastes the script output in their vote message,
it points to the exact version of the tool that was used, assuming
people check the script's digest when running it.
-Bertrand
[1] https://github.com/apache/openwhisk-release/blob/master/tools/rcverify.sh
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Radu,
In concept I agree, no good reason to have multiple tools to do similar
things. I wonder if we just fork off the release validation process into a
separate script vs the Java-based processes. I'm just not convinced of the
value re-implementing in Java brings vs a few simple bash commands. Let me
poke around and see if I can come up with something.
Thanks,
Dan
On Fri, Aug 23, 2019 at 6:31 AM Radu Cotescu <ra...@apache.org> wrote:
> Hello,
>
> I’m +1 for any tool that reduces the number of manual steps I have to do
> when performing, validating or promoting a release.
>
> I have one question though: couldn’t we push these features into the
> Apache Sling Committer CLI tool [0]? That tool is already based on a
> minimalistic Docker image with an embedded jlink-customised JRE. Sensible
> information is provided from the outside anyways (think Apache user and
> password, JIRA user and password), so if we’re thinking about the keys file
> we could do something similar.
>
> The only reason for asking such a question is that it would be nice to
> consolidate all CLI checks into one tool.
>
> Thanks,
> Radu
>
> [0] - https://github.com/apache/sling-org-apache-sling-committer-cli <
> https://github.com/apache/sling-org-apache-sling-committer-cli>
Re: Release Validator Docker Image
Posted by Radu Cotescu <ra...@apache.org>.
Hello,
I’m +1 for any tool that reduces the number of manual steps I have to do when performing, validating or promoting a release.
I have one question though: couldn’t we push these features into the Apache Sling Committer CLI tool [0]? That tool is already based on a minimalistic Docker image with an embedded jlink-customised JRE. Sensible information is provided from the outside anyways (think Apache user and password, JIRA user and password), so if we’re thinking about the keys file we could do something similar.
The only reason for asking such a question is that it would be nice to consolidate all CLI checks into one tool.
Thanks,
Radu
[0] - https://github.com/apache/sling-org-apache-sling-committer-cli <https://github.com/apache/sling-org-apache-sling-committer-cli>
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Nice idea! I'm thinking we could have a default set of commands, e.g. run
the check staged release and CI status and then a command line option to
attach the symlinks for more.
So I've got an early crack at it in the whiteboard. I am a bit confused
about the behavior of the ci check script you provided. For example,
checking release 2115, it returned an error code as follows:
Status for org.apache.sling.commons.scheduler 2.7.4: pending
See
https://github.com/apache/sling-org-apache-sling-commons-scheduler/commits/org.apache.sling.commons.scheduler-2.7.4
for details
Exit code: 129
When I checked the status it didn't seem very illuminating, so I checked
the build status which was successful:
https://builds.apache.org/blue/organizations/jenkins/Sling%2Fsling-org-apache-sling-commons-scheduler/activity
And Sonar:
https://sonarcloud.io/dashboard?id=apache_sling-org-apache-sling-commons-scheduler
Is there just something about that project that's not configured yet? Is
there some way from the Github API to get the cause?
On Thu, Aug 22, 2019 at 4:04 AM Robert Munteanu <ro...@apache.org> wrote:
> I was recently thinking about the way we check releases. We currently
> have a number of scripts:
>
> - check_staged_release.sh
> - check_release_matches_tag.sh
>
> Also I have my own scripts to check the CI status and build locally.
> You have proposed a release validator docker image. I would suggest
> tying thet up together in the following manner.
>
> 1. Have a top-level check_sling_release.sh script which drives the
> whole process
>
> 2. Create checks-enabled and checks-available directories
>
> The checks-enabled will include
>
> 0001_check_staged_release.sh
>
> The checks-available will contain all other checks, e.g.
>
> 0002_check_release_matches_tag.sh
> 0003_check_validator_image.sh
> 0004_check_ci_status.sh
> 0005_check_local_build.sh
>
> Enabling a certain check checks to run then means creating (for
> example) a symlink from checks-
> available/0002_check_release_matches_tag.sh to checks-enabled
>
> This way we can both accumulate more checks and allow flexibility in
> which checks are selected.
>
> As for the validator image, we can have in the tooling-release
> repository or a different one, I don't have a strong preference either
> way.
>
> Thanks,
> Robert
>
>
> On Tue, 2019-08-06 at 09:07 -0400, Daniel Klco wrote:
> > Sorry for the slow turnaround on this. I updated to use OpenJDK and
> > rejected the idea of running the integration tests as it makes the
> > execution time extremely slow. Is this something we'd be interested
> > in
> > moving out of the Whiteboard? Unfortunately, it does have quite a
> > different
> > structure than the committer-cli due to being mostly bash scripts.
> >
> >
> > On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <ro...@apache.org>
> > wrote:
> >
> > > On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > > > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > > > hand, it'd
> > > > be better not not require the local RPM, on the other hand, to
> > > > support
> > > > validation it'd be good to be able (I would think) to compile
> > > > against
> > > > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> > > >
> > > > I guess it's a question to the overall team, is there any need to
> > > > validate
> > > > against OracleJDK or should I just base this off OpenDJK?
> > >
> > > I always used OpenJDK.
> > >
> > > Robert
> > >
> > > > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <
> > > > rombert@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Dan,
> > > > >
> > > > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > > > I've also been working on a docker image to validate and
> > > > > > build
> > > > > > releases:
> > > > > >
> > > > > >
> > >
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > > > > It's relatively easy to use
> > > > >
> > > > > That looks nice! One question - do you need to have the Java
> > > > > RPM
> > > > > locally? If yes, have you considered using one of the OpenJDK
> > > > > docker
> > > > > images, which already do this for you?
> > > > >
> > > > > https://hub.docker.com/_/openjdk
> > > > >
> > > > > As a side note, I have some local scripts which do the
> > > > > following:
> > > > >
> > > > > - check the status of a release on Github
> > > > > - builds a reactor of the whole
> > > > >
> > > > > I am a bit lazy but I'll attach them here, in case you want to
> > > > > pick
> > > > > something up for the validator docker image feel free :-)
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Robert
> > > > >
>
>
Re: Release Validator Docker Image
Posted by Robert Munteanu <ro...@apache.org>.
I was recently thinking about the way we check releases. We currently
have a number of scripts:
- check_staged_release.sh
- check_release_matches_tag.sh
Also I have my own scripts to check the CI status and build locally.
You have proposed a release validator docker image. I would suggest
tying thet up together in the following manner.
1. Have a top-level check_sling_release.sh script which drives the
whole process
2. Create checks-enabled and checks-available directories
The checks-enabled will include
0001_check_staged_release.sh
The checks-available will contain all other checks, e.g.
0002_check_release_matches_tag.sh
0003_check_validator_image.sh
0004_check_ci_status.sh
0005_check_local_build.sh
Enabling a certain check checks to run then means creating (for
example) a symlink from checks-
available/0002_check_release_matches_tag.sh to checks-enabled
This way we can both accumulate more checks and allow flexibility in
which checks are selected.
As for the validator image, we can have in the tooling-release
repository or a different one, I don't have a strong preference either
way.
Thanks,
Robert
On Tue, 2019-08-06 at 09:07 -0400, Daniel Klco wrote:
> Sorry for the slow turnaround on this. I updated to use OpenJDK and
> rejected the idea of running the integration tests as it makes the
> execution time extremely slow. Is this something we'd be interested
> in
> moving out of the Whiteboard? Unfortunately, it does have quite a
> different
> structure than the committer-cli due to being mostly bash scripts.
>
>
> On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <ro...@apache.org>
> wrote:
>
> > On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > > hand, it'd
> > > be better not not require the local RPM, on the other hand, to
> > > support
> > > validation it'd be good to be able (I would think) to compile
> > > against
> > > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> > >
> > > I guess it's a question to the overall team, is there any need to
> > > validate
> > > against OracleJDK or should I just base this off OpenDJK?
> >
> > I always used OpenJDK.
> >
> > Robert
> >
> > > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <
> > > rombert@apache.org>
> > > wrote:
> > >
> > > > Hi Dan,
> > > >
> > > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > > I've also been working on a docker image to validate and
> > > > > build
> > > > > releases:
> > > > >
> > > > >
> > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > > > It's relatively easy to use
> > > >
> > > > That looks nice! One question - do you need to have the Java
> > > > RPM
> > > > locally? If yes, have you considered using one of the OpenJDK
> > > > docker
> > > > images, which already do this for you?
> > > >
> > > > https://hub.docker.com/_/openjdk
> > > >
> > > > As a side note, I have some local scripts which do the
> > > > following:
> > > >
> > > > - check the status of a release on Github
> > > > - builds a reactor of the whole
> > > >
> > > > I am a bit lazy but I'll attach them here, in case you want to
> > > > pick
> > > > something up for the validator docker image feel free :-)
> > > >
> > > > Thanks,
> > > >
> > > > Robert
> > > >
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Sorry for the slow turnaround on this. I updated to use OpenJDK and
rejected the idea of running the integration tests as it makes the
execution time extremely slow. Is this something we'd be interested in
moving out of the Whiteboard? Unfortunately, it does have quite a different
structure than the committer-cli due to being mostly bash scripts.
On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <ro...@apache.org> wrote:
> On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > hand, it'd
> > be better not not require the local RPM, on the other hand, to
> > support
> > validation it'd be good to be able (I would think) to compile against
> > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> >
> > I guess it's a question to the overall team, is there any need to
> > validate
> > against OracleJDK or should I just base this off OpenDJK?
>
> I always used OpenJDK.
>
> Robert
>
> >
> > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <ro...@apache.org>
> > wrote:
> >
> > > Hi Dan,
> > >
> > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > I've also been working on a docker image to validate and build
> > > > releases:
> > > >
> > > >
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > >
> > > > It's relatively easy to use
> > >
> > > That looks nice! One question - do you need to have the Java RPM
> > > locally? If yes, have you considered using one of the OpenJDK
> > > docker
> > > images, which already do this for you?
> > >
> > > https://hub.docker.com/_/openjdk
> > >
> > > As a side note, I have some local scripts which do the following:
> > >
> > > - check the status of a release on Github
> > > - builds a reactor of the whole
> > >
> > > I am a bit lazy but I'll attach them here, in case you want to pick
> > > something up for the validator docker image feel free :-)
> > >
> > > Thanks,
> > >
> > > Robert
> > >
>
>
Re: Release Validator Docker Image
Posted by Robert Munteanu <ro...@apache.org>.
On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> Yeah, I've gone back and forth regarding the JDK RPM. On the one
> hand, it'd
> be better not not require the local RPM, on the other hand, to
> support
> validation it'd be good to be able (I would think) to compile against
> multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
>
> I guess it's a question to the overall team, is there any need to
> validate
> against OracleJDK or should I just base this off OpenDJK?
I always used OpenJDK.
Robert
>
> On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <ro...@apache.org>
> wrote:
>
> > Hi Dan,
> >
> > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > I've also been working on a docker image to validate and build
> > > releases:
> > >
> > > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > >
> > > It's relatively easy to use
> >
> > That looks nice! One question - do you need to have the Java RPM
> > locally? If yes, have you considered using one of the OpenJDK
> > docker
> > images, which already do this for you?
> >
> > https://hub.docker.com/_/openjdk
> >
> > As a side note, I have some local scripts which do the following:
> >
> > - check the status of a release on Github
> > - builds a reactor of the whole
> >
> > I am a bit lazy but I'll attach them here, in case you want to pick
> > something up for the validator docker image feel free :-)
> >
> > Thanks,
> >
> > Robert
> >
Re: Release Validator Docker Image
Posted by Daniel Klco <da...@gmail.com>.
Yeah, I've gone back and forth regarding the JDK RPM. On the one hand, it'd
be better not not require the local RPM, on the other hand, to support
validation it'd be good to be able (I would think) to compile against
multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
I guess it's a question to the overall team, is there any need to validate
against OracleJDK or should I just base this off OpenDJK?
On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <ro...@apache.org> wrote:
> Hi Dan,
>
> On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > I've also been working on a docker image to validate and build
> > releases:
> >
> > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> >
> > It's relatively easy to use
>
> That looks nice! One question - do you need to have the Java RPM
> locally? If yes, have you considered using one of the OpenJDK docker
> images, which already do this for you?
>
> https://hub.docker.com/_/openjdk
>
> As a side note, I have some local scripts which do the following:
>
> - check the status of a release on Github
> - builds a reactor of the whole
>
> I am a bit lazy but I'll attach them here, in case you want to pick
> something up for the validator docker image feel free :-)
>
> Thanks,
>
> Robert
>
Re: Release Validator Docker Image
Posted by Robert Munteanu <ro...@apache.org>.
Hi Dan,
On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> I've also been working on a docker image to validate and build
> releases:
>
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
>
> It's relatively easy to use
That looks nice! One question - do you need to have the Java RPM
locally? If yes, have you considered using one of the OpenJDK docker
images, which already do this for you?
https://hub.docker.com/_/openjdk
As a side note, I have some local scripts which do the following:
- check the status of a release on Github
- builds a reactor of the whole
I am a bit lazy but I'll attach them here, in case you want to pick
something up for the validator docker image feel free :-)
Thanks,
Robert