You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Luca Toscano <to...@gmail.com> on 2024/01/07 14:38:56 UTC
Installed plugin for pipelines
Hi folks,
just letting you know that I have installed the Pipelines plugin in
the Jenkins' master, to test if we can define our jobs in there and
avoid the Matrix reloaded plugin. It would be also very nice to have
all pipelines checked out in the BigTop's repo eventually.
I'll open a jira to track all the work after some quick tests :)
Let me know if you see anything weird, I've just restarted Jenkins to
pick up the new plugin.
Luca
Re: Installed plugin for pipelines
Posted by Luca Toscano <to...@gmail.com>.
Hi folks,
getting back to the subject if anyone is interested in giving some
feedback. In https://issues.apache.org/jira/browse/BIGTOP-4058 there
are some examples of Jenkins pipelines that I have created to see if
we can get rid of the Matrix reloaded plugin, but before proceeding
I'd need to know if it is something that people would use/like or not
:) The main use case for Matrix reloaded seems to be (IIUC) having a
way to selectively trigger builds and smoke tests while doing a
release (to bypass transient failures to specific components that
would require to trigger all the builds again). In theory we could do
the same with pipelines, but they are a little less flexible for the
moment (at least from a UI perspective). Getting rid of the Matrix
plugin would avoid some old/unsecure code and it would align us with
more recent Jenkins standards.
Lemme know!
Thanks in advance,
Luca
On Mon, Jan 15, 2024 at 12:06 PM Luca Toscano <to...@gmail.com> wrote:
>
> As promised: https://issues.apache.org/jira/browse/BIGTOP-4058
>
> I started adding some jobs to show the idea, feedback welcome :)
>
> Luca
>
> On Thu, Jan 11, 2024 at 3:56 PM Luca Toscano <to...@gmail.com> wrote:
> >
> > Nevermind I fixed the apt issue, but the current problem is that the
> > COMPONENTS of the smoke test cannot be grouped together. IIUC we want
> > to be able to kick off a run with COMPONENTS set to all or a subset of
> > daemons (yarn/hdfs/etc..), that will bootstrap the hadoop cluster via
> > Docker. I'll open a task and work on it, hopefully there is an easy
> > solution :)
> >
> > On Thu, Jan 11, 2024 at 12:11 PM Luca Toscano <to...@gmail.com> wrote:
> > >
> > > I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/
> > >
> > > It currently doesn't work since there seems to be an error when apt
> > > runs inside the Docker container, but it is a generic script to give
> > > you the idea. Release version and branch have a separate parameter, at
> > > the moment we have 3.2.0 and 3.3.0, so a single script could in theory
> > > be used multiple times with minimal changes (IIUC at the moment for
> > > every release we create new smoke tests etc..).\
> > >
> > > To test it:
> > >
> > > 1) Run build
> > > 2) Open the build run's "Console output"
> > > 3) Hit "Input Requested" and you'll see a series of dropdown options.
> > > Once you select the right ones, the build should start.
> > >
> > > If you want to see an example:
> > > https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console
> > >
> > > This is just an idea to gather feedback, it can surely be improved,
> > > but it is a starting point :)
> > >
> > > Thanks!
> > >
> > > Luca
> > >
> > > On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <to...@gmail.com> wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > I created https://ci.bigtop.apache.org/job/test-elukey as an example
> > > > of how Jenkins pipelines can be used to replace the Matrix Reloaded
> > > > plugin. The prototype is limited for the moment to:
> > > > - It uses scripted pipelines, not declarative pipelines, since the
> > > > former is more flexible in my opinion.
> > > > - It mimics the Smoke Tests for Debian OSes, used in our release
> > > > process IIUC (like
> > > > https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/)
> > > > - It only echoes values, no build is done.
> > > > - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT
> > > > at runtime), but the inputs need to be added in the build's console
> > > > output rather than having specific "Build with params" in Jenkins UI.
> > > > This is probably something that we can circumvent, but the current
> > > > solution avoids to copy/paste values multiple times etc..
> > > > - Compared to Matrix reloaded, the flexibility of selecting dynamic
> > > > parameters is less, but it should be sufficient for the release use
> > > > case.
> > > >
> > > > We'd need to deprecate Matrix Reloaded
> > > > (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was
> > > > abandoned since a long time ago, and no more fixes are provided from
> > > > upstream.
> > > >
> > > > Do you think that this process is viable? If so we could use pipelines
> > > > as experimental step during the next release, to see if they can be
> > > > adopted. Let me know your thoughts and if I am missing something big
> > > > (probably happening, apologies in advance).
> > > >
> > > > Thanks!
> > > >
> > > > Luca
> > > >
> > > > On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <to...@gmail.com> wrote:
> > > > >
> > > > > Hi folks,
> > > > >
> > > > > just letting you know that I have installed the Pipelines plugin in
> > > > > the Jenkins' master, to test if we can define our jobs in there and
> > > > > avoid the Matrix reloaded plugin. It would be also very nice to have
> > > > > all pipelines checked out in the BigTop's repo eventually.
> > > > >
> > > > > I'll open a jira to track all the work after some quick tests :)
> > > > >
> > > > > Let me know if you see anything weird, I've just restarted Jenkins to
> > > > > pick up the new plugin.
> > > > >
> > > > > Luca
Re: Installed plugin for pipelines
Posted by Luca Toscano <to...@gmail.com>.
As promised: https://issues.apache.org/jira/browse/BIGTOP-4058
I started adding some jobs to show the idea, feedback welcome :)
Luca
On Thu, Jan 11, 2024 at 3:56 PM Luca Toscano <to...@gmail.com> wrote:
>
> Nevermind I fixed the apt issue, but the current problem is that the
> COMPONENTS of the smoke test cannot be grouped together. IIUC we want
> to be able to kick off a run with COMPONENTS set to all or a subset of
> daemons (yarn/hdfs/etc..), that will bootstrap the hadoop cluster via
> Docker. I'll open a task and work on it, hopefully there is an easy
> solution :)
>
> On Thu, Jan 11, 2024 at 12:11 PM Luca Toscano <to...@gmail.com> wrote:
> >
> > I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/
> >
> > It currently doesn't work since there seems to be an error when apt
> > runs inside the Docker container, but it is a generic script to give
> > you the idea. Release version and branch have a separate parameter, at
> > the moment we have 3.2.0 and 3.3.0, so a single script could in theory
> > be used multiple times with minimal changes (IIUC at the moment for
> > every release we create new smoke tests etc..).\
> >
> > To test it:
> >
> > 1) Run build
> > 2) Open the build run's "Console output"
> > 3) Hit "Input Requested" and you'll see a series of dropdown options.
> > Once you select the right ones, the build should start.
> >
> > If you want to see an example:
> > https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console
> >
> > This is just an idea to gather feedback, it can surely be improved,
> > but it is a starting point :)
> >
> > Thanks!
> >
> > Luca
> >
> > On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <to...@gmail.com> wrote:
> > >
> > > Hi folks,
> > >
> > > I created https://ci.bigtop.apache.org/job/test-elukey as an example
> > > of how Jenkins pipelines can be used to replace the Matrix Reloaded
> > > plugin. The prototype is limited for the moment to:
> > > - It uses scripted pipelines, not declarative pipelines, since the
> > > former is more flexible in my opinion.
> > > - It mimics the Smoke Tests for Debian OSes, used in our release
> > > process IIUC (like
> > > https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/)
> > > - It only echoes values, no build is done.
> > > - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT
> > > at runtime), but the inputs need to be added in the build's console
> > > output rather than having specific "Build with params" in Jenkins UI.
> > > This is probably something that we can circumvent, but the current
> > > solution avoids to copy/paste values multiple times etc..
> > > - Compared to Matrix reloaded, the flexibility of selecting dynamic
> > > parameters is less, but it should be sufficient for the release use
> > > case.
> > >
> > > We'd need to deprecate Matrix Reloaded
> > > (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was
> > > abandoned since a long time ago, and no more fixes are provided from
> > > upstream.
> > >
> > > Do you think that this process is viable? If so we could use pipelines
> > > as experimental step during the next release, to see if they can be
> > > adopted. Let me know your thoughts and if I am missing something big
> > > (probably happening, apologies in advance).
> > >
> > > Thanks!
> > >
> > > Luca
> > >
> > > On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <to...@gmail.com> wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > just letting you know that I have installed the Pipelines plugin in
> > > > the Jenkins' master, to test if we can define our jobs in there and
> > > > avoid the Matrix reloaded plugin. It would be also very nice to have
> > > > all pipelines checked out in the BigTop's repo eventually.
> > > >
> > > > I'll open a jira to track all the work after some quick tests :)
> > > >
> > > > Let me know if you see anything weird, I've just restarted Jenkins to
> > > > pick up the new plugin.
> > > >
> > > > Luca
Re: Installed plugin for pipelines
Posted by Luca Toscano <to...@gmail.com>.
Nevermind I fixed the apt issue, but the current problem is that the
COMPONENTS of the smoke test cannot be grouped together. IIUC we want
to be able to kick off a run with COMPONENTS set to all or a subset of
daemons (yarn/hdfs/etc..), that will bootstrap the hadoop cluster via
Docker. I'll open a task and work on it, hopefully there is an easy
solution :)
On Thu, Jan 11, 2024 at 12:11 PM Luca Toscano <to...@gmail.com> wrote:
>
> I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/
>
> It currently doesn't work since there seems to be an error when apt
> runs inside the Docker container, but it is a generic script to give
> you the idea. Release version and branch have a separate parameter, at
> the moment we have 3.2.0 and 3.3.0, so a single script could in theory
> be used multiple times with minimal changes (IIUC at the moment for
> every release we create new smoke tests etc..).\
>
> To test it:
>
> 1) Run build
> 2) Open the build run's "Console output"
> 3) Hit "Input Requested" and you'll see a series of dropdown options.
> Once you select the right ones, the build should start.
>
> If you want to see an example:
> https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console
>
> This is just an idea to gather feedback, it can surely be improved,
> but it is a starting point :)
>
> Thanks!
>
> Luca
>
> On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <to...@gmail.com> wrote:
> >
> > Hi folks,
> >
> > I created https://ci.bigtop.apache.org/job/test-elukey as an example
> > of how Jenkins pipelines can be used to replace the Matrix Reloaded
> > plugin. The prototype is limited for the moment to:
> > - It uses scripted pipelines, not declarative pipelines, since the
> > former is more flexible in my opinion.
> > - It mimics the Smoke Tests for Debian OSes, used in our release
> > process IIUC (like
> > https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/)
> > - It only echoes values, no build is done.
> > - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT
> > at runtime), but the inputs need to be added in the build's console
> > output rather than having specific "Build with params" in Jenkins UI.
> > This is probably something that we can circumvent, but the current
> > solution avoids to copy/paste values multiple times etc..
> > - Compared to Matrix reloaded, the flexibility of selecting dynamic
> > parameters is less, but it should be sufficient for the release use
> > case.
> >
> > We'd need to deprecate Matrix Reloaded
> > (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was
> > abandoned since a long time ago, and no more fixes are provided from
> > upstream.
> >
> > Do you think that this process is viable? If so we could use pipelines
> > as experimental step during the next release, to see if they can be
> > adopted. Let me know your thoughts and if I am missing something big
> > (probably happening, apologies in advance).
> >
> > Thanks!
> >
> > Luca
> >
> > On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <to...@gmail.com> wrote:
> > >
> > > Hi folks,
> > >
> > > just letting you know that I have installed the Pipelines plugin in
> > > the Jenkins' master, to test if we can define our jobs in there and
> > > avoid the Matrix reloaded plugin. It would be also very nice to have
> > > all pipelines checked out in the BigTop's repo eventually.
> > >
> > > I'll open a jira to track all the work after some quick tests :)
> > >
> > > Let me know if you see anything weird, I've just restarted Jenkins to
> > > pick up the new plugin.
> > >
> > > Luca
Re: Installed plugin for pipelines
Posted by Luca Toscano <to...@gmail.com>.
I have also created: https://ci.bigtop.apache.org/job/pipeline-smoke-test/
It currently doesn't work since there seems to be an error when apt
runs inside the Docker container, but it is a generic script to give
you the idea. Release version and branch have a separate parameter, at
the moment we have 3.2.0 and 3.3.0, so a single script could in theory
be used multiple times with minimal changes (IIUC at the moment for
every release we create new smoke tests etc..).\
To test it:
1) Run build
2) Open the build run's "Console output"
3) Hit "Input Requested" and you'll see a series of dropdown options.
Once you select the right ones, the build should start.
If you want to see an example:
https://ci.bigtop.apache.org/job/pipeline-smoke-test/5/console
This is just an idea to gather feedback, it can surely be improved,
but it is a starting point :)
Thanks!
Luca
On Wed, Jan 10, 2024 at 9:44 PM Luca Toscano <to...@gmail.com> wrote:
>
> Hi folks,
>
> I created https://ci.bigtop.apache.org/job/test-elukey as an example
> of how Jenkins pipelines can be used to replace the Matrix Reloaded
> plugin. The prototype is limited for the moment to:
> - It uses scripted pipelines, not declarative pipelines, since the
> former is more flexible in my opinion.
> - It mimics the Smoke Tests for Debian OSes, used in our release
> process IIUC (like
> https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/)
> - It only echoes values, no build is done.
> - It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT
> at runtime), but the inputs need to be added in the build's console
> output rather than having specific "Build with params" in Jenkins UI.
> This is probably something that we can circumvent, but the current
> solution avoids to copy/paste values multiple times etc..
> - Compared to Matrix reloaded, the flexibility of selecting dynamic
> parameters is less, but it should be sufficient for the release use
> case.
>
> We'd need to deprecate Matrix Reloaded
> (https://plugins.jenkins.io/matrix-reloaded/) since the plugin was
> abandoned since a long time ago, and no more fixes are provided from
> upstream.
>
> Do you think that this process is viable? If so we could use pipelines
> as experimental step during the next release, to see if they can be
> adopted. Let me know your thoughts and if I am missing something big
> (probably happening, apologies in advance).
>
> Thanks!
>
> Luca
>
> On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <to...@gmail.com> wrote:
> >
> > Hi folks,
> >
> > just letting you know that I have installed the Pipelines plugin in
> > the Jenkins' master, to test if we can define our jobs in there and
> > avoid the Matrix reloaded plugin. It would be also very nice to have
> > all pipelines checked out in the BigTop's repo eventually.
> >
> > I'll open a jira to track all the work after some quick tests :)
> >
> > Let me know if you see anything weird, I've just restarted Jenkins to
> > pick up the new plugin.
> >
> > Luca
Re: Installed plugin for pipelines
Posted by Luca Toscano <to...@gmail.com>.
Hi folks,
I created https://ci.bigtop.apache.org/job/test-elukey as an example
of how Jenkins pipelines can be used to replace the Matrix Reloaded
plugin. The prototype is limited for the moment to:
- It uses scripted pipelines, not declarative pipelines, since the
former is more flexible in my opinion.
- It mimics the Smoke Tests for Debian OSes, used in our release
process IIUC (like
https://ci.bigtop.apache.org/view/3.2.0-smoke-tests/job/Bigtop-3.2.0-debian-10-smoke-tests/1/)
- It only echoes values, no build is done.
- It allows dynamic parameters (namely to restrict OS/ARCH/COMPONENT
at runtime), but the inputs need to be added in the build's console
output rather than having specific "Build with params" in Jenkins UI.
This is probably something that we can circumvent, but the current
solution avoids to copy/paste values multiple times etc..
- Compared to Matrix reloaded, the flexibility of selecting dynamic
parameters is less, but it should be sufficient for the release use
case.
We'd need to deprecate Matrix Reloaded
(https://plugins.jenkins.io/matrix-reloaded/) since the plugin was
abandoned since a long time ago, and no more fixes are provided from
upstream.
Do you think that this process is viable? If so we could use pipelines
as experimental step during the next release, to see if they can be
adopted. Let me know your thoughts and if I am missing something big
(probably happening, apologies in advance).
Thanks!
Luca
On Sun, Jan 7, 2024 at 3:38 PM Luca Toscano <to...@gmail.com> wrote:
>
> Hi folks,
>
> just letting you know that I have installed the Pipelines plugin in
> the Jenkins' master, to test if we can define our jobs in there and
> avoid the Matrix reloaded plugin. It would be also very nice to have
> all pipelines checked out in the BigTop's repo eventually.
>
> I'll open a jira to track all the work after some quick tests :)
>
> Let me know if you see anything weird, I've just restarted Jenkins to
> pick up the new plugin.
>
> Luca