You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@plc4x.apache.org by Sebastian Rühl <se...@googlemail.com> on 2018/03/01 07:23:35 UTC
Re: Jenkins: Moving from scripted Pipeline to declarative Pipeline
Hi Chris,
Looks good.
Regarding the problem with the missing graphviz. We could do a stash after the build and lets the site stages run all on the git-websites node.
What do you think?
Sebastian
> Am 28.02.2018 um 12:00 schrieb Christofer Dutz <ch...@c-ware.de>:
>
> Ok I think I have a valid solution
>
> I left things pretty much the way they were, but excluded most of the plugin executions that would usually kick in in the deploy step.
> Now it seems as only the steps we want are executed. I definitely like this now as now we are really only deploying things if all tests pass.
> So now untested code in an upstream module can't break everything downstream.
>
> Unfortunately we additionally had a second problem: Fontawesome 5 was released recently and therefore the download of 4.7 was discontinued and this caused the build to fail. I removed the dependency to that as I could not trace down any usage of this anywhere in the generated site. This also greatly reduces the amount of stuff having to be deployed.
>
> Please have a look at my changes.
>
> Chris
>
>
>
>
> Am 28.02.18, 09:22 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Ok,
>
> bringing back the discussion I just had with Sebastian (has problems accessing the mailing list at his current customer).
>
> I see the following problems with the current build pipeline:
> 1. The second "deploy" execution runs the build and tests double
> 2. The site should be generated on the same node as the tests are run (currently not happening when using the Site-Generation Job, but should be)
>
> The reason Sebastian split up "Build & Test" and "Deploy" was to prevent things being deployed if the tests fail, which makes sense.
>
> I would propose to re-unite the build, test, site-generation and sonar-analysis to one step (Maven is built to handle all of this, I see no benefit in splitting this up).
> Additionally we activate the experimental "deployAtEnd" switch of the deploy plugin. This should only deploy anything if all modules builds are finished and the deploy phase of the last module is executed.
>
> What do you think?
>
> Chris
>
>
>
>
> Am 27.02.18, 18:43 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> By the way,
>
> as it seems there is no clean solution to be able to run the rawsockets tests without a dedicated VM, I ordered one from infra today.
> This VM can be configured to our needs and we no longer have to jump any hoops. This VM will be a Jenkins node (probably labeled plc4x) which is only available to our projects build jobs. If we need anything installed, we can add it to the puppet scripts for it and the changes will get deployed.
>
> I really hope this way we'll get rid of some of the most annoying problems we were fighting with and keep the build simple at the same time.
>
> The option of building docker images for running integration-tests sounded extremely complicated from my point of view.
>
> The option to not execute the raw socket tests is not really an option, I think as I think the main CI server should execute all tests and not just a subset.
>
> Chris
>
>
>
> Am 27.02.18, 18:15 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Hi Sebastian,
>
> I just noticed two things with the new piepeline. It now correctly is able to deploy the site from the main build, so that's cool.
> Unfortunately it seems that now the site is no longer able to generate the images in via blockdiag etc.
>
> Also I did notice that the split between build and deploy has a very undesirable side effect:
>
> - Build does a "clean install", which does clean, compile, test, integration-test
> - Deploy does a "deploy", which does compile, test, integration-test, deploy
>
> So most of the build is executed twice, we should eliminate that, however I don't think Maven likes this sort of split-up and I doubt a "mvn deploy:deploy" will automatically know what to deploy.
>
> Right now I would suggest to bring the deploy back to the "build master" as it's only deployed when built on master.
>
>
> Chris
>
>
>
> Am 24.02.18, 16:55 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Hi Sebastian,
>
> I finally managed to have my flu-infected brain have a look at the changes. I do agree it l looks a lot cleaner this way. I also like the way the Jenkins jobs now look.
> Hopefully now we could also do the following things:
> - Add a "site-deploy" step that is only executed on the master branch, which deploys the site. In order for this to work, we would need to enforce this part being executed on a node tagged with "git-websites" as only these can actually push to the git-site branch.
>
> The second thing is a little more complex:
> - In order to run the RawSocket tests, we need to have libpcap with setuid root or have to execute the tests as root. Both is not possible on the normal Jenkins VMs for good reasons. There would however be an option to fire up a Docker container and run the tests inside that. Unfortunately I need a lot of help with setting up something like that (Ideally this would be handled in Maven and not in the Jenkinsfile, as this way the test-results and coverages are cleanly integrated with the rest of the build)
>
> Seems the docker-maven-plugin from Fabric8 would be a possible solution, but I remember that Fabric8 has been cancelled by RedHat ... would it still be ok to use that plugin?
> http://info.michael-simons.eu/2016/08/25/integration-testing-with-docker-and-maven/
>
> Chris
>
>
>
> Am 22.02.18, 13:02 schrieb "Sebastian Rühl" <se...@googlemail.com>:
>
> Hi Together,
>
> Today I found the time to move from scripted pipeline to declarative pipeline.
> This opens a much better integration into Jenkins and makes the Jenkinsfile much more readable.
>
> Small changes to before:
> - The build stage is split into two (Build, Build master) where I converted the if statement to stage conditions.
> - Testsresults are reported to Jenkins and can now easily be seen in the build view
> - Toolmanagement is moved to native integration (so no need for explicitly defining envs anyomore)
>
> What I want to do next is to use the native sonar integration. This will then use the sonar configuration defined in Jenkins and ad a sonar link into Jenkins. Then we also could define a build-breaker for quality gates in the future.
>
> Sebastian
>
>
>
>
>
>
>
>
>
Re: Jenkins: Moving from scripted Pipeline to declarative Pipeline
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Sebastian,
I think we should simply wait till we have our vm and keep the current build till that's done.
Even if the stash option seems to work, it does offload the work to the git system. Shelving the entire workspace doesn't feel right. Don't want to waste ASF's resources.
Chris
Outlook for Android<https://aka.ms/ghei36> herunterladen
Von: Sebastian Rühl
Gesendet: Donnerstag, 1. März, 08:23
Betreff: Re: Jenkins: Moving from scripted Pipeline to declarative Pipeline
An: dev@plc4x.apache.org
Hi Chris, Looks good. Regarding the problem with the missing graphviz. We could do a stash after the build and lets the site stages run all on the git-websites node. What do you think? Sebastian > Am 28.02.2018 um 12:00 schrieb Christofer Dutz : > > Ok I think I have a valid solution > > I left things pretty much the way they were, but excluded most of the plugin executions that would usually kick in in the deploy step. > Now it seems as only the steps we want are executed. I definitely like this now as now we are really only deploying things if all tests pass. > So now untested code in an upstream module can't break everything downstream. > > Unfortunately we additionally had a second problem: Fontawesome 5 was released recently and therefore the download of 4.7 was discontinued and this caused the build to fail. I removed the dependency to that as I could not trace down any usage of this anywhere in the generated site. This also greatly reduces the amount of stuff having to be deployed. > > Please have a look at my changes. > > Chris > > > > > Am 28.02.18, 09:22 schrieb "Christofer Dutz" : > > Ok, > > bringing back the discussion I just had with Sebastian (has problems accessing the mailing list at his current customer). > > I see the following problems with the current build pipeline: > 1. The second "deploy" execution runs the build and tests double > 2. The site should be generated on the same node as the tests are run (currently not happening when using the Site-Generation Job, but should be) > > The reason Sebastian split up "Build & Test" and "Deploy" was to prevent things being deployed if the tests fail, which makes sense. > > I would propose to re-unite the build, test, site-generation and sonar-analysis to one step (Maven is built to handle all of this, I see no benefit in splitting this up). > Additionally we activate the experimental "deployAtEnd" switch of the deploy plugin. This should only deploy anything if all modules builds are finished and the deploy phase of the last module is executed. > > What do you think? > > Chris > > > > > Am 27.02.18, 18:43 schrieb "Christofer Dutz" : > > By the way, > > as it seems there is no clean solution to be able to run the rawsockets tests without a dedicated VM, I ordered one from infra today. > This VM can be configured to our needs and we no longer have to jump any hoops. This VM will be a Jenkins node (probably labeled plc4x) which is only available to our projects build jobs. If we need anything installed, we can add it to the puppet scripts for it and the changes will get deployed. > > I really hope this way we'll get rid of some of the most annoying problems we were fighting with and keep the build simple at the same time. > > The option of building docker images for running integration-tests sounded extremely complicated from my point of view. > > The option to not execute the raw socket tests is not really an option, I think as I think the main CI server should execute all tests and not just a subset. > > Chris > > > > Am 27.02.18, 18:15 schrieb "Christofer Dutz" : > > Hi Sebastian, > > I just noticed two things with the new piepeline. It now correctly is able to deploy the site from the main build, so that's cool. > Unfortunately it seems that now the site is no longer able to generate the images in via blockdiag etc. > > Also I did notice that the split between build and deploy has a very undesirable side effect: > > - Build does a "clean install", which does clean, compile, test, integration-test > - Deploy does a "deploy", which does compile, test, integration-test, deploy > > So most of the build is executed twice, we should eliminate that, however I don't think Maven likes this sort of split-up and I doubt a "mvn deploy:deploy" will automatically know what to deploy. > > Right now I would suggest to bring the deploy back to the "build master" as it's only deployed when built on master. > > > Chris > > > > Am 24.02.18, 16:55 schrieb "Christofer Dutz" : > > Hi Sebastian, > > I finally managed to have my flu-infected brain have a look at the changes. I do agree it l looks a lot cleaner this way. I also like the way the Jenkins jobs now look. > Hopefully now we could also do the following things: > - Add a "site-deploy" step that is only executed on the master branch, which deploys the site. In order for this to work, we would need to enforce this part being executed on a node tagged with "git-websites" as only these can actually push to the git-site branch. > > The second thing is a little more complex: > - In order to run the RawSocket tests, we need to have libpcap with setuid root or have to execute the tests as root. Both is not possible on the normal Jenkins VMs for good reasons. There would however be an option to fire up a Docker container and run the tests inside that. Unfortunately I need a lot of help with setting up something like that (Ideally this would be handled in Maven and not in the Jenkinsfile, as this way the test-results and coverages are cleanly integrated with the rest of the build) > > Seems the docker-maven-plugin from Fabric8 would be a possible solution, but I remember that Fabric8 has been cancelled by RedHat ... would it still be ok to use that plugin? > http://info.michael-simons.eu/2016/08/25/integration-testing-with-docker-and-maven/ > > Chris > > > > Am 22.02.18, 13:02 schrieb "Sebastian Rühl" : > > Hi Together, > > Today I found the time to move from scripted pipeline to declarative pipeline. > This opens a much better integration into Jenkins and makes the Jenkinsfile much more readable. > > Small changes to before: > - The build stage is split into two (Build, Build master) where I converted the if statement to stage conditions. > - Testsresults are reported to Jenkins and can now easily be seen in the build view > - Toolmanagement is moved to native integration (so no need for explicitly defining envs anyomore) > > What I want to do next is to use the native sonar integration. This will then use the sonar configuration defined in Jenkins and ad a sonar link into Jenkins. Then we also could define a build-breaker for quality gates in the future. > > Sebastian > > > > > > > > >