You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Gabriel Bräscher <ga...@gmail.com> on 2020/04/14 20:44:24 UTC
Re: Jenkins
Hello Alex, hope that you are well.
Thank you very much for the reply and sorry for the (very) late reply.
I just got back to the Jenkins situation (we will release a new CloudStack
version soon).
I totally understand that we cannot sign the packages in an automated way,
that is far from secure.
Regarding the package building, can you please advise if it is possible to
configure a job on builds.apache.org?
I know that there are companies using CloudStack able to donate slaves to
help running the jobs packaging. If there is a way to contribute, please
let me know.
Additionally, I have access to create jobs, if that is ok I wold be
configuring a few jobs to help the project with CI/packaging and tests. We
already have a lot of help with current job, however on some cases we need
to manually test the PR, and having a way of sharing the build on the
development phase would be a big plus.
Best regards,
Gabriel.
Em qui., 12 de dez. de 2019 às 13:33, Alex Harui <ah...@adobe.com>
escreveu:
> AIUI, you can automate everything except the verification and PGP
> signing. The building and packaging doesn't have to be done on computers
> under the RM's control, but before the RM applies his/her PGP signature,
> the RM must convince themselves that the artifacts they are signing are
> correct and safe. IMO the way to do that is to download the artifacts to a
> computer under the RM's control that has the RM's PGP key on it, and then
> run some sort of tests before signing. For the source package, it can be
> as simple as checking out the tag into another folder and doing a diff.
> For binaries it is much harder, but with the trend towards reproducible
> binaries, I believe it is now practical.
>
> But then the RM has a pile of signed artifacts on some computer that has
> to be uploaded to the distribution servers. A script can help with that,
> though.
>
> HTH,
> -Alex
>
> On 12/12/19, 2:55 AM, "Gabriel Beims Bräscher" <ga...@apache.org>
> wrote:
>
> Hello,
>
> I am an Apache CloudStack PMC/Committer.
>
> One of the goals that we at CloudStack have is to automatically build
> packages and update our mirrors whenever a new release is launched.
> I worked as release manager for CloudStack 4.12.0.0 (non-LTS) and
> assisted on 4.13.0.0 (LTS), In both cases I executed the building
> process manually (build all the deb and rpm packages and make them
> available at the ACS repository mirror). It would be great to make it
> automated.
>
> With that in mind I am wondering If is there a way to set up Jenkins.
> Additionally, I work at PCextreme and as a cloud provider we would be
> happy to donate resources (virtual machine(s)), if necessary to run
> such
> jobs.
>
> Thanks for all the help!
> Best regards,
> Gabriel.
>
> --
> Gabriel Beims Bräscher
> Apache CloudStack Committer/PMC
> The Apache Software Foundation
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=rJarxm3rQh2Kh9oirMkQWWISQ4WquxJrO6Jr%2F2chWak%3D&reserved=0
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=jCkeUb804f7jntV9hCgVCa7cIS6MjKuN%2B4NLqgOyzIc%3D&reserved=0
>
>
>
>
Re: Jenkins
Posted by Vladimir Sitnikov <si...@gmail.com>.
Alex>Ideas for improvement are welcome
Please don't get me wrong, however, I have the following feedback for
Apache JMeter (see [1]):
Milamber>Before, with the ant release way, ~21 command lines to execute to
Milamber>prepare the Vote email
Milamber>After, with the gradle way, 1 command line(*) : ./gradlew
prepareVote
Milamber>-Prc=4 -Pasf -PuseGpgCmd
I would recommend giving Gradle a try.
It makes it easy to create and maintain release-like automation sequences.
You might notice that JMeter is Ant -> Gradle migration, however, I have a
similar improvement for Apache Calcite with Maven -> Gradle.
[1]:
https://lists.apache.org/thread.html/c08780952b7fbd71eccff717cb494562723e11dc99ea1b4bb39af461%40%3Cdev.jmeter.apache.org%3E
Vladimir
Re: Jenkins
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Hmm. I didn't think there were "sides".
The fact that Apache Royale has 100 steps has nothing to do with using a CI server to create release artifacts. It is just because Royale not only distributes maven artifacts from 3 repos, but also two flavors of legacy artifacts are created and distributed via dist.a.o and NPM as well. Our RMs have 100 steps even if they use a local machine. The Maven artifacts are created in the first 20-ish commands.
I certainly hope I didn't claim that Royale's process is perfect and a reference model. I've been trying to say that it now seems practical to use a CI server in the release process because there are reproducible binary plugins for Maven. Royale is in fact doing that, but because reproducible binaries are relatively new, I would not say that what we have is a reference model. We just have something that works.
To provide a bit more detail on what we've got working: where an RM on a local machine typically does a release:branch, release:prepare, and release:perform, on a CI server, I think there needs to be more work between each step. So you can't have a single CI job that runs all three steps, you have to have a CI job for each step, and do some work between steps and manually launch the next step.
That's because the CI server has to run release:branch with pushChanges=false because there shouldn't be any committer credentials on the CI server. The RM has to login and push the changes by supplying username and password. Then the CI server can run release:prepare, also with pushChanges=false. And again, the RM has to login and push changes and the tag.
Then when the CI server runs release:perform, the flag skipGPG=true has to be set. And we also add an altDeploymentDirectory. That's because release:perform not only generates the binaries, but it signs them and uploads them to the staging server. The CI server has to skip signing since the keys can't be stored on the server. So what our CI job does is redirect the binary artifacts to an altDeploymentDirectory and use Jenkins to archive the altDeploymentDIrectory as build artifacts. Then the RM downloads the artifacts to a local machine, gets the source package and builds it, then compares the jars that were built with the ones in the altDeploymentDIrectory. If they match, the RM can sign the artifacts and then use Maven's Wagon to upload them to the staging server.
Ideas for improvement are welcome. This is just the first thing that came to mind that worked and seemed secure and conformed to Apache's rules/policies. As you can see the RM still has to build the source code locally, all voters need to be able to build the source anyway, and the RM can theoretically save some time by skipping some/all tests because they were run on the CI server and the binaries are bit-for-bit identical.
HTH,
-Alex
On 4/14/20, 11:59 PM, "Christofer Dutz" <ch...@c-ware.de> wrote:
Hi Gabriel,
I just have to jump in here ... as I saw you were only getting feedback from one side.
I did see that Cloudstack generally seems to be built with Maven, so I can give some input
There is absolutely nothing wrong with building dev snapshots on a CI server ... that's one of the things they were built for.
However when talking about releases the situation is different. Because you need a real persons credentials for:
- Pushing to Git repos
- Signing artifacts
- Deploying artifact to any form of remote repos
So these are the parts you can't automate and a RM will have to jump in manually.
So what's typically a 3 command release process on an ordinary maven project:
- branch
- prepare
- release
Has to be split up and divided into a sequence of steps where some can be done "automatic" (by clicking a button) and some have to be done manually.
I would also strongly suggest not to use the Apache Royale release process as a reference. It was intended to simplify things and currently is a process of about 100 steps that have to be executed in sequence ... some by clicking a button ... some by logging in on the CI server (which would probably be a problem on shared ASF build nodes).
I doubt this is something you would like to do. I. strongly suggest not to.
So please keep in mind that you can't have any non-technical user credentials on a shared CI server node ... you could probably do that if you have a dedicated Release Manager machine (Sort of you start it up, provide the credentials needed and after the Release the VM is discarded) but it's totally out of the question for a shared machine. And as Apache doesn't have technical users outside of infra, well I guess the only options you have are:
- dedicated RM machine
- very lengthy release process with lots of manual steps
Chris
Am 14.04.20, 22:44 schrieb "Gabriel Bräscher" <ga...@gmail.com>:
Hello Alex, hope that you are well.
Thank you very much for the reply and sorry for the (very) late reply.
I just got back to the Jenkins situation (we will release a new CloudStack
version soon).
I totally understand that we cannot sign the packages in an automated way,
that is far from secure.
Regarding the package building, can you please advise if it is possible to
configure a job on builds.apache.org?
I know that there are companies using CloudStack able to donate slaves to
help running the jobs packaging. If there is a way to contribute, please
let me know.
Additionally, I have access to create jobs, if that is ok I wold be
configuring a few jobs to help the project with CI/packaging and tests. We
already have a lot of help with current job, however on some cases we need
to manually test the PR, and having a way of sharing the build on the
development phase would be a big plus.
Best regards,
Gabriel.
Em qui., 12 de dez. de 2019 às 13:33, Alex Harui <ah...@adobe.com>
escreveu:
> AIUI, you can automate everything except the verification and PGP
> signing. The building and packaging doesn't have to be done on computers
> under the RM's control, but before the RM applies his/her PGP signature,
> the RM must convince themselves that the artifacts they are signing are
> correct and safe. IMO the way to do that is to download the artifacts to a
> computer under the RM's control that has the RM's PGP key on it, and then
> run some sort of tests before signing. For the source package, it can be
> as simple as checking out the tag into another folder and doing a diff.
> For binaries it is much harder, but with the trend towards reproducible
> binaries, I believe it is now practical.
>
> But then the RM has a pile of signed artifacts on some computer that has
> to be uploaded to the distribution servers. A script can help with that,
> though.
>
> HTH,
> -Alex
>
> On 12/12/19, 2:55 AM, "Gabriel Beims Bräscher" <ga...@apache.org>
> wrote:
>
> Hello,
>
> I am an Apache CloudStack PMC/Committer.
>
> One of the goals that we at CloudStack have is to automatically build
> packages and update our mirrors whenever a new release is launched.
> I worked as release manager for CloudStack 4.12.0.0 (non-LTS) and
> assisted on 4.13.0.0 (LTS), In both cases I executed the building
> process manually (build all the deb and rpm packages and make them
> available at the ACS repository mirror). It would be great to make it
> automated.
>
> With that in mind I am wondering If is there a way to set up Jenkins.
> Additionally, I work at PCextreme and as a cloud provider we would be
> happy to donate resources (virtual machine(s)), if necessary to run
> such
> jobs.
>
> Thanks for all the help!
> Best regards,
> Gabriel.
>
> --
> Gabriel Beims Bräscher
> Apache CloudStack Committer/PMC
> The Apache Software Foundation
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7Ce72f016ca83a409a6a7808d7e10a8345%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637225307639333642&sdata=gf6rjeVocgFiOqP8tyGjJ9uuWlt93v%2F%2BCiyYM3SdXj8%3D&reserved=0
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7Ce72f016ca83a409a6a7808d7e10a8345%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637225307639333642&sdata=VHfFZMF6CnMPScHhE5cii7%2Fw8vgToMamnP8Y4CWAUNc%3D&reserved=0
>
>
>
>
Re: Jenkins
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Gabriel,
I just have to jump in here ... as I saw you were only getting feedback from one side.
I did see that Cloudstack generally seems to be built with Maven, so I can give some input
There is absolutely nothing wrong with building dev snapshots on a CI server ... that's one of the things they were built for.
However when talking about releases the situation is different. Because you need a real persons credentials for:
- Pushing to Git repos
- Signing artifacts
- Deploying artifact to any form of remote repos
So these are the parts you can't automate and a RM will have to jump in manually.
So what's typically a 3 command release process on an ordinary maven project:
- branch
- prepare
- release
Has to be split up and divided into a sequence of steps where some can be done "automatic" (by clicking a button) and some have to be done manually.
I would also strongly suggest not to use the Apache Royale release process as a reference. It was intended to simplify things and currently is a process of about 100 steps that have to be executed in sequence ... some by clicking a button ... some by logging in on the CI server (which would probably be a problem on shared ASF build nodes).
I doubt this is something you would like to do. I. strongly suggest not to.
So please keep in mind that you can't have any non-technical user credentials on a shared CI server node ... you could probably do that if you have a dedicated Release Manager machine (Sort of you start it up, provide the credentials needed and after the Release the VM is discarded) but it's totally out of the question for a shared machine. And as Apache doesn't have technical users outside of infra, well I guess the only options you have are:
- dedicated RM machine
- very lengthy release process with lots of manual steps
Chris
Am 14.04.20, 22:44 schrieb "Gabriel Bräscher" <ga...@gmail.com>:
Hello Alex, hope that you are well.
Thank you very much for the reply and sorry for the (very) late reply.
I just got back to the Jenkins situation (we will release a new CloudStack
version soon).
I totally understand that we cannot sign the packages in an automated way,
that is far from secure.
Regarding the package building, can you please advise if it is possible to
configure a job on builds.apache.org?
I know that there are companies using CloudStack able to donate slaves to
help running the jobs packaging. If there is a way to contribute, please
let me know.
Additionally, I have access to create jobs, if that is ok I wold be
configuring a few jobs to help the project with CI/packaging and tests. We
already have a lot of help with current job, however on some cases we need
to manually test the PR, and having a way of sharing the build on the
development phase would be a big plus.
Best regards,
Gabriel.
Em qui., 12 de dez. de 2019 às 13:33, Alex Harui <ah...@adobe.com>
escreveu:
> AIUI, you can automate everything except the verification and PGP
> signing. The building and packaging doesn't have to be done on computers
> under the RM's control, but before the RM applies his/her PGP signature,
> the RM must convince themselves that the artifacts they are signing are
> correct and safe. IMO the way to do that is to download the artifacts to a
> computer under the RM's control that has the RM's PGP key on it, and then
> run some sort of tests before signing. For the source package, it can be
> as simple as checking out the tag into another folder and doing a diff.
> For binaries it is much harder, but with the trend towards reproducible
> binaries, I believe it is now practical.
>
> But then the RM has a pile of signed artifacts on some computer that has
> to be uploaded to the distribution servers. A script can help with that,
> though.
>
> HTH,
> -Alex
>
> On 12/12/19, 2:55 AM, "Gabriel Beims Bräscher" <ga...@apache.org>
> wrote:
>
> Hello,
>
> I am an Apache CloudStack PMC/Committer.
>
> One of the goals that we at CloudStack have is to automatically build
> packages and update our mirrors whenever a new release is launched.
> I worked as release manager for CloudStack 4.12.0.0 (non-LTS) and
> assisted on 4.13.0.0 (LTS), In both cases I executed the building
> process manually (build all the deb and rpm packages and make them
> available at the ACS repository mirror). It would be great to make it
> automated.
>
> With that in mind I am wondering If is there a way to set up Jenkins.
> Additionally, I work at PCextreme and as a cloud provider we would be
> happy to donate resources (virtual machine(s)), if necessary to run
> such
> jobs.
>
> Thanks for all the help!
> Best regards,
> Gabriel.
>
> --
> Gabriel Beims Bräscher
> Apache CloudStack Committer/PMC
> The Apache Software Foundation
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=rJarxm3rQh2Kh9oirMkQWWISQ4WquxJrO6Jr%2F2chWak%3D&reserved=0
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=jCkeUb804f7jntV9hCgVCa7cIS6MjKuN%2B4NLqgOyzIc%3D&reserved=0
>
>
>
>
Re: Jenkins
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Hello Gabriel,
I’m just a user of the service and not part of the group that runs it. You can read more about this service here: https://cwiki.apache.org/confluence/display/INFRA/Jenkins
I’m not knowledgeable enough about the service to say what you can’t do. I was only saying explaining how my project builds releases on shared machines, which we do on machines outside of builds.a.o. Because the machines in build.a.o are shared with other projects there may be technical or social limits on how many resources your jobs can use. We don’t have that problem on our own machines, but then we have to spend time keeping them running.
Good luck,
-Alex
From: Gabriel Bräscher <ga...@gmail.com>
Date: Tuesday, April 14, 2020 at 1:44 PM
To: Alex Harui <ah...@adobe.com>
Cc: "builds@apache.org" <bu...@apache.org>, "gabriel@apache.org" <ga...@apache.org>
Subject: Re: Jenkins
Hello Alex, hope that you are well.
Thank you very much for the reply and sorry for the (very) late reply.
I just got back to the Jenkins situation (we will release a new CloudStack version soon).
I totally understand that we cannot sign the packages in an automated way, that is far from secure.
Regarding the package building, can you please advise if it is possible to configure a job on builds.apache.org<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbuilds.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7Cd27f05db0b1341951b7008d7e0b4a6ab%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637224938804391565&sdata=AykhNSy0fDVWWTNMM6PPblEPLlVPgMBm5oPgs32rQvo%3D&reserved=0>?
I know that there are companies using CloudStack able to donate slaves to help running the jobs packaging. If there is a way to contribute, please let me know.
Additionally, I have access to create jobs, if that is ok I wold be configuring a few jobs to help the project with CI/packaging and tests. We already have a lot of help with current job, however on some cases we need to manually test the PR, and having a way of sharing the build on the development phase would be a big plus.
Best regards,
Gabriel.
Em qui., 12 de dez. de 2019 às 13:33, Alex Harui <ah...@adobe.com>> escreveu:
AIUI, you can automate everything except the verification and PGP signing. The building and packaging doesn't have to be done on computers under the RM's control, but before the RM applies his/her PGP signature, the RM must convince themselves that the artifacts they are signing are correct and safe. IMO the way to do that is to download the artifacts to a computer under the RM's control that has the RM's PGP key on it, and then run some sort of tests before signing. For the source package, it can be as simple as checking out the tag into another folder and doing a diff. For binaries it is much harder, but with the trend towards reproducible binaries, I believe it is now practical.
But then the RM has a pile of signed artifacts on some computer that has to be uploaded to the distribution servers. A script can help with that, though.
HTH,
-Alex
On 12/12/19, 2:55 AM, "Gabriel Beims Bräscher" <ga...@apache.org>> wrote:
Hello,
I am an Apache CloudStack PMC/Committer.
One of the goals that we at CloudStack have is to automatically build
packages and update our mirrors whenever a new release is launched.
I worked as release manager for CloudStack 4.12.0.0 (non-LTS) and
assisted on 4.13.0.0 (LTS), In both cases I executed the building
process manually (build all the deb and rpm packages and make them
available at the ACS repository mirror). It would be great to make it
automated.
With that in mind I am wondering If is there a way to set up Jenkins.
Additionally, I work at PCextreme and as a cloud provider we would be
happy to donate resources (virtual machine(s)), if necessary to run such
jobs.
Thanks for all the help!
Best regards,
Gabriel.
--
Gabriel Beims Bräscher
Apache CloudStack Committer/PMC
The Apache Software Foundation
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=rJarxm3rQh2Kh9oirMkQWWISQ4WquxJrO6Jr%2F2chWak%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7Cd27f05db0b1341951b7008d7e0b4a6ab%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637224938804391565&sdata=xORuFOkLP%2BekgWN5lfC5dqDBlnsA%2B2JZGXmZ3uzP1mc%3D&reserved=0>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7C2776a1f593ae42724ff208d77ef1bc38%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C637117449021619621&sdata=jCkeUb804f7jntV9hCgVCa7cIS6MjKuN%2B4NLqgOyzIc%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudstack.apache.org%2F&data=02%7C01%7Caharui%40adobe.com%7Cd27f05db0b1341951b7008d7e0b4a6ab%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C637224938804401559&sdata=53VoVt9rTqtTuHzCECGMMcxM0CZb8o73etAAdPPRVzM%3D&reserved=0>