You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Jim Apple <jb...@cloudera.com> on 2016/12/13 21:51:23 UTC
Jenkins jobs to verify patches
Before Impala joined the ASF, code reviews had to go through a
"verification" step in which a Jenkins jobs downloaded the patch, ran
all tests and replied back to Jenkins that everything was OK. That
Jenkins job ran on Cloudera infrastructure and could not be accessed,
even in a read-only way, by people outside of Cloudera.
To follow the Apache way, I am laboring to replace that with a Jenkins
server that can be used by any authorized person and read by any
person. It is at http://jenkins.impala.io:8080. It is able to verify
patches just like the old Jenkins machine. A few remaining questions:
1. What should the prerequisites be to turning off the Cloudera-only
Jenkins verification path?
2. Who should be able to run jobs on jenkins.impala.io? Some
possibilities: Committers only, anyone who asks, PMC members only,
contributors who ask after submitting 5 patches. Higher bars lead to
less likelihood of abuse, lower ones to easier contributions from
newbies.
My proposal is this:
1. We should turn off the CLoudera-only Jenkins verification path
January 2. jenkins.impala.io is in pretty good shape, and we can make
further improvements as needed. For instance, it took me five minutes
just now to cut down on the not-so-interesting debug output from the
main verification job,
http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
2. Everyone with five patches can request a login.
I'm not married to these ideas, but I wanted to provide a jumping-off
point for discussion.
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
As discussed in the thread, requiring the use of gerrit provides an
auditing method - the code must at least be posted there, so if
something goes haywire we can see the code that caused it.
However, you can post a draft if the code is not yet ready for review.
the Jenkins job can test your draft.
On Thu, Jan 5, 2017 at 4:13 AM, Lars Volker <lv...@cloudera.com> wrote:
> Is it possible without much effort to make this work on github urls, too?
> Currently our policy seems to be that we pick up reviews as soon as they're
> posted, so there doesn't seem to be a way for non-Cloudera non-committers
> to run tests on Jenkins before submitting a change for review.
>
> On Thu, Jan 5, 2017 at 3:02 AM, Jim Apple <jb...@cloudera.com> wrote:
>
>> http://jenkins.impala.io:8080/job/pre-review-test/ is now open for
>> business.
>>
>> On Tue, Jan 3, 2017 at 3:53 PM, Jim Apple <jb...@cloudera.com> wrote:
>> > Given that sentiment, I'll plan on opening up a limited-parallelism
>> > job for anonymous users that only allows gerrit patches.
>> >
>> > I will also plan the switchover to happen on January 9, six days from
>> > today. On that day, I'll turn off the Cloudera-VPN-gated pre-merge
>> > job.
>> >
>> > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
>> wrote:
>> >> Got it.
>> >>
>> >> I think I'd probably be more in favour of handing out login credential
>> to
>> >> contributors on demand (e.g. by mailing a list) rather than having open
>> >> access, just so we have a clearer idea of who's using it. I don't have a
>> >> strong objection to the alternative.
>> >>
>> >> On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
>> wrote:
>> >>
>> >>> > How isolated is the Jenkins instance?
>> >>>
>> >>> As far as I know, the workers have little access to the coordinator.
>> See
>> >>> here:
>> >>>
>> >>> https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
>> Master+Access+Control
>> >>>
>> >>> This flag is on and there are no whitelisted exceptions.
>> >>>
>> >>> > Does the jenkins user have many privileges on the VM?
>> >>>
>> >>> They have passwordless sudo on the worker
>> >>>
>> >>> > Could it simply wipe
>> >>> > out the job history to destroy the trail?
>> >>>
>> >>> Job history is stored on the coordinator.
>> >>>
>> >>> > Jenkins also presumably has
>> >>> > credentials to make at least some changes to gerrit - are those
>> >>> privileges
>> >>> > restrictive enough that it couldn't cause problems there too?
>> >>>
>> >>> Those are stored only on the coordinator and cannot be used by the
>> slaves.
>> >>>
>>
Re: Jenkins jobs to verify patches
Posted by Lars Volker <lv...@cloudera.com>.
Is it possible without much effort to make this work on github urls, too?
Currently our policy seems to be that we pick up reviews as soon as they're
posted, so there doesn't seem to be a way for non-Cloudera non-committers
to run tests on Jenkins before submitting a change for review.
On Thu, Jan 5, 2017 at 3:02 AM, Jim Apple <jb...@cloudera.com> wrote:
> http://jenkins.impala.io:8080/job/pre-review-test/ is now open for
> business.
>
> On Tue, Jan 3, 2017 at 3:53 PM, Jim Apple <jb...@cloudera.com> wrote:
> > Given that sentiment, I'll plan on opening up a limited-parallelism
> > job for anonymous users that only allows gerrit patches.
> >
> > I will also plan the switchover to happen on January 9, six days from
> > today. On that day, I'll turn off the Cloudera-VPN-gated pre-merge
> > job.
> >
> > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
> wrote:
> >> Got it.
> >>
> >> I think I'd probably be more in favour of handing out login credential
> to
> >> contributors on demand (e.g. by mailing a list) rather than having open
> >> access, just so we have a clearer idea of who's using it. I don't have a
> >> strong objection to the alternative.
> >>
> >> On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
> wrote:
> >>
> >>> > How isolated is the Jenkins instance?
> >>>
> >>> As far as I know, the workers have little access to the coordinator.
> See
> >>> here:
> >>>
> >>> https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> Master+Access+Control
> >>>
> >>> This flag is on and there are no whitelisted exceptions.
> >>>
> >>> > Does the jenkins user have many privileges on the VM?
> >>>
> >>> They have passwordless sudo on the worker
> >>>
> >>> > Could it simply wipe
> >>> > out the job history to destroy the trail?
> >>>
> >>> Job history is stored on the coordinator.
> >>>
> >>> > Jenkins also presumably has
> >>> > credentials to make at least some changes to gerrit - are those
> >>> privileges
> >>> > restrictive enough that it couldn't cause problems there too?
> >>>
> >>> Those are stored only on the coordinator and cannot be used by the
> slaves.
> >>>
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
http://jenkins.impala.io:8080/job/pre-review-test/ is now open for business.
On Tue, Jan 3, 2017 at 3:53 PM, Jim Apple <jb...@cloudera.com> wrote:
> Given that sentiment, I'll plan on opening up a limited-parallelism
> job for anonymous users that only allows gerrit patches.
>
> I will also plan the switchover to happen on January 9, six days from
> today. On that day, I'll turn off the Cloudera-VPN-gated pre-merge
> job.
>
> On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com> wrote:
>> Got it.
>>
>> I think I'd probably be more in favour of handing out login credential to
>> contributors on demand (e.g. by mailing a list) rather than having open
>> access, just so we have a clearer idea of who's using it. I don't have a
>> strong objection to the alternative.
>>
>> On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com> wrote:
>>
>>> > How isolated is the Jenkins instance?
>>>
>>> As far as I know, the workers have little access to the coordinator. See
>>> here:
>>>
>>> https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+Master+Access+Control
>>>
>>> This flag is on and there are no whitelisted exceptions.
>>>
>>> > Does the jenkins user have many privileges on the VM?
>>>
>>> They have passwordless sudo on the worker
>>>
>>> > Could it simply wipe
>>> > out the job history to destroy the trail?
>>>
>>> Job history is stored on the coordinator.
>>>
>>> > Jenkins also presumably has
>>> > credentials to make at least some changes to gerrit - are those
>>> privileges
>>> > restrictive enough that it couldn't cause problems there too?
>>>
>>> Those are stored only on the coordinator and cannot be used by the slaves.
>>>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
Given that sentiment, I'll plan on opening up a limited-parallelism
job for anonymous users that only allows gerrit patches.
I will also plan the switchover to happen on January 9, six days from
today. On that day, I'll turn off the Cloudera-VPN-gated pre-merge
job.
On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com> wrote:
> Got it.
>
> I think I'd probably be more in favour of handing out login credential to
> contributors on demand (e.g. by mailing a list) rather than having open
> access, just so we have a clearer idea of who's using it. I don't have a
> strong objection to the alternative.
>
> On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com> wrote:
>
>> > How isolated is the Jenkins instance?
>>
>> As far as I know, the workers have little access to the coordinator. See
>> here:
>>
>> https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+Master+Access+Control
>>
>> This flag is on and there are no whitelisted exceptions.
>>
>> > Does the jenkins user have many privileges on the VM?
>>
>> They have passwordless sudo on the worker
>>
>> > Could it simply wipe
>> > out the job history to destroy the trail?
>>
>> Job history is stored on the coordinator.
>>
>> > Jenkins also presumably has
>> > credentials to make at least some changes to gerrit - are those
>> privileges
>> > restrictive enough that it couldn't cause problems there too?
>>
>> Those are stored only on the coordinator and cannot be used by the slaves.
>>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
Kudu runs tests preemptively when a gerrit patch first appears.
On Dec 14, 2016 9:58 AM, "Tim Armstrong" <ta...@cloudera.com> wrote:
> I mean the contributor could email an email address (e.g. a mailing list)
> asking for credentials and we could email them privately.
>
> Do we know what other Apache projects do for situations like this?
>
> On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com> wrote:
>
> > Can you clarify the "credentials by mailing list" approach?
> >
> > If we send out the credentials on a public list, it's pretty close to
> open
> > access.
> >
> > If we send out credentials to contributors privately, we have an
> additional
> > hurdle to contributions.
> >
> > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > Got it.
> > >
> > > I think I'd probably be more in favour of handing out login credential
> to
> > > contributors on demand (e.g. by mailing a list) rather than having
> open
> > > access, just so we have a clearer idea of who's using it. I don't have
> a
> > > strong objection to the alternative.
> > >
> > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
> wrote:
> > >
> > > > > How isolated is the Jenkins instance?
> > > >
> > > > As far as I know, the workers have little access to the coordinator.
> > See
> > > > here:
> > > >
> > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > Master+Access+Control
> > > >
> > > > This flag is on and there are no whitelisted exceptions.
> > > >
> > > > > Does the jenkins user have many privileges on the VM?
> > > >
> > > > They have passwordless sudo on the worker
> > > >
> > > > > Could it simply wipe
> > > > > out the job history to destroy the trail?
> > > >
> > > > Job history is stored on the coordinator.
> > > >
> > > > > Jenkins also presumably has
> > > > > credentials to make at least some changes to gerrit - are those
> > > > privileges
> > > > > restrictive enough that it couldn't cause problems there too?
> > > >
> > > > Those are stored only on the coordinator and cannot be used by the
> > > slaves.
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada
<bh...@cloudera.com> wrote:
> Just wondering why we can't link Jenkins authentication with gerrit login
Gerrit login is just github login, and so provides little in the way
of access control.
Additionally, Jenkins has weak support for other authentication
methods, often through outdated or flaky plugins.
> isolate Jenkins to only run
> code thats approved (+2ed) over gerrit.
That's a different question, and one we could probably do.
> With this, any new contributor
> (whoever has signed up on gerrit) can have access to the jenkins box
It doesn't quite work like that - if we wanted to do that, each
Jenkins job would have to authenticate that the patch had been +2ed.
That would also prevent speculative testing of non-+2ed patches by
committers, which is a legitimate use case.
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
That seems reasonable. However, I would like to consider that and
non-security-impacting usability improvements separately, as a
different issue, and focus this thread on the two items from the OP:
1. when we can turn off Cloudera's internal Jenkins, 2. Who should
have access to what jobs and in what configuration.
On Wed, Dec 14, 2016 at 4:07 PM, Bharath Vissapragada
<bh...@cloudera.com> wrote:
> Yes, in terms of security, it would be the same as a separate Jenkins
> login. But this would be much easier to use/manage IMO, both for
> contributors as well as admins.
>
> On Wed, Dec 14, 2016 at 2:00 PM, Jim Apple <jb...@cloudera.com> wrote:
>
>> I do not understand the proposed security benefit of using github for
>> user authentication over using Jenkins permissions directly. Keep in
>> mind that preventing non-+2ed commits from being run is an orthogonal
>> question - this can be configured with any user authentication method.
>>
>> On Wed, Dec 14, 2016 at 1:52 PM, Bharath Vissapragada
>> <bh...@cloudera.com> wrote:
>> > That was just for configuring the gerrit trigger. Regarding auth
>> > integration, I read this page. Does it not work?
>> >
>> > https://wiki.jenkins-ci.org/display/JENKINS/GitHub+OAuth+
>> Plugin#GitHubOAuthPlugin-AboutGitHubAuthenticationPlugin
>> >
>> > On Wed, Dec 14, 2016 at 1:50 PM, Jim Apple <jb...@cloudera.com> wrote:
>> >
>> >> I think you're misunderstanding that plugin. It does not associate
>> >> Jenkins login with gerrit login.
>> >>
>> >> On Wed, Dec 14, 2016 at 1:45 PM, Bharath Vissapragada
>> >> <bh...@cloudera.com> wrote:
>> >> > On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com>
>> >> wrote:
>> >> >
>> >> >> That integration sounds like a great idea to me.
>> >> >>
>> >> >> Just to clarify the purpose: We'd like external contributors to be
>> able
>> >> to
>> >> >> run private build+test runs during development, and not just run GVO
>> >> once
>> >> >> the patch has +2. The hard part is gettig the patch to +2 in the
>> first
>> >> >> place and we've seen instances where relying on local test runs only
>> >> can be
>> >> >> difficult.
>> >> >>
>> >> >> For example, contributors could upload a draft to gerrit and run a
>> >> private
>> >> >> build to fix problems before publishing the patch fir review.
>> >> >>
>> >> >
>> >> > As per this [1] link, I think that can be configured. I haven't tried
>> it
>> >> > myself, but from the looks of it, it seems plausible.
>> >> >
>> >> > [1]
>> >> > https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+
>> >> Trigger#GerritTrigger-TriggerConfiguration
>> >> >
>> >> >
>> >> >>
>> >> >> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
>> >> >> bharathv@cloudera.com
>> >> >> > wrote:
>> >> >>
>> >> >> > Just wondering why we can't link Jenkins authentication with gerrit
>> >> login
>> >> >> > in this case instead of having two separate login credentials. That
>> >> way
>> >> >> we
>> >> >> > can retain the audit trail of the jobs and also isolate Jenkins to
>> >> only
>> >> >> run
>> >> >> > code thats approved (+2ed) over gerrit. With this, any new
>> contributor
>> >> >> > (whoever has signed up on gerrit) can have access to the jenkins
>> box
>> >> and
>> >> >> we
>> >> >> > can be sure that they only run the stuff that is approved by
>> >> committers.
>> >> >> > Thoughts?
>> >> >> >
>> >> >> > * I'm not totally sure if such an integration is possible but I
>> did a
>> >> >> quick
>> >> >> > search and I got a feeling that shouldn't be difficult.
>> >> >> >
>> >> >> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
>> >> >> > tbobrovytsky@cloudera.com> wrote:
>> >> >> >
>> >> >> > > I would be more in favor of starting with open access instead of
>> >> having
>> >> >> > to
>> >> >> > > hand out credentials. It's both less work for us and it makes it
>> >> easier
>> >> >> > to
>> >> >> > > contribute. If we notice that this is not working well, or gets
>> >> abused,
>> >> >> > we
>> >> >> > > can switch to what Tim is suggesting. Also, we should be able to
>> see
>> >> >> who
>> >> >> > is
>> >> >> > > using our Jenkins by looking at Gerrit (because the patch must be
>> >> >> > uploaded
>> >> >> > > to Gerrit before starting a build).
>> >> >> > >
>> >> >> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <
>> alex.behm@cloudera.com
>> >> >
>> >> >> > > wrote:
>> >> >> > >
>> >> >> > > > I'm fine with Tim's approach, but it does add some friction to
>> >> >> > > > contributions.
>> >> >> > > >
>> >> >> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
>> >> >> > tarmstrong@cloudera.com>
>> >> >> > > > wrote:
>> >> >> > > >
>> >> >> > > > > I mean the contributor could email an email address (e.g. a
>> >> mailing
>> >> >> > > list)
>> >> >> > > > > asking for credentials and we could email them privately.
>> >> >> > > > >
>> >> >> > > > > Do we know what other Apache projects do for situations like
>> >> this?
>> >> >> > > > >
>> >> >> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <
>> >> alex.behm@cloudera.com
>> >> >> >
>> >> >> > > > wrote:
>> >> >> > > > >
>> >> >> > > > > > Can you clarify the "credentials by mailing list" approach?
>> >> >> > > > > >
>> >> >> > > > > > If we send out the credentials on a public list, it's
>> pretty
>> >> >> close
>> >> >> > to
>> >> >> > > > > open
>> >> >> > > > > > access.
>> >> >> > > > > >
>> >> >> > > > > > If we send out credentials to contributors privately, we
>> have
>> >> an
>> >> >> > > > > additional
>> >> >> > > > > > hurdle to contributions.
>> >> >> > > > > >
>> >> >> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
>> >> >> > > > tarmstrong@cloudera.com>
>> >> >> > > > > > wrote:
>> >> >> > > > > >
>> >> >> > > > > > > Got it.
>> >> >> > > > > > >
>> >> >> > > > > > > I think I'd probably be more in favour of handing out
>> login
>> >> >> > > > credential
>> >> >> > > > > to
>> >> >> > > > > > > contributors on demand (e.g. by mailing a list) rather
>> than
>> >> >> > having
>> >> >> > > > > open
>> >> >> > > > > > > access, just so we have a clearer idea of who's using
>> it. I
>> >> >> don't
>> >> >> > > > have
>> >> >> > > > > a
>> >> >> > > > > > > strong objection to the alternative.
>> >> >> > > > > > >
>> >> >> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
>> >> >> jbapple@cloudera.com
>> >> >> > >
>> >> >> > > > > wrote:
>> >> >> > > > > > >
>> >> >> > > > > > > > > How isolated is the Jenkins instance?
>> >> >> > > > > > > >
>> >> >> > > > > > > > As far as I know, the workers have little access to the
>> >> >> > > > coordinator.
>> >> >> > > > > > See
>> >> >> > > > > > > > here:
>> >> >> > > > > > > >
>> >> >> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
>> >> >> > > > > > > Master+Access+Control
>> >> >> > > > > > > >
>> >> >> > > > > > > > This flag is on and there are no whitelisted
>> exceptions.
>> >> >> > > > > > > >
>> >> >> > > > > > > > > Does the jenkins user have many privileges on the VM?
>> >> >> > > > > > > >
>> >> >> > > > > > > > They have passwordless sudo on the worker
>> >> >> > > > > > > >
>> >> >> > > > > > > > > Could it simply wipe
>> >> >> > > > > > > > > out the job history to destroy the trail?
>> >> >> > > > > > > >
>> >> >> > > > > > > > Job history is stored on the coordinator.
>> >> >> > > > > > > >
>> >> >> > > > > > > > > Jenkins also presumably has
>> >> >> > > > > > > > > credentials to make at least some changes to gerrit -
>> >> are
>> >> >> > those
>> >> >> > > > > > > > privileges
>> >> >> > > > > > > > > restrictive enough that it couldn't cause problems
>> there
>> >> >> too?
>> >> >> > > > > > > >
>> >> >> > > > > > > > Those are stored only on the coordinator and cannot be
>> >> used
>> >> >> by
>> >> >> > > the
>> >> >> > > > > > > slaves.
>> >> >> > > > > > > >
>> >> >> > > > > > >
>> >> >> > > > > >
>> >> >> > > > >
>> >> >> > > >
>> >> >> > >
>> >> >> >
>> >> >>
>> >>
>>
Re: Jenkins jobs to verify patches
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Yes, in terms of security, it would be the same as a separate Jenkins
login. But this would be much easier to use/manage IMO, both for
contributors as well as admins.
On Wed, Dec 14, 2016 at 2:00 PM, Jim Apple <jb...@cloudera.com> wrote:
> I do not understand the proposed security benefit of using github for
> user authentication over using Jenkins permissions directly. Keep in
> mind that preventing non-+2ed commits from being run is an orthogonal
> question - this can be configured with any user authentication method.
>
> On Wed, Dec 14, 2016 at 1:52 PM, Bharath Vissapragada
> <bh...@cloudera.com> wrote:
> > That was just for configuring the gerrit trigger. Regarding auth
> > integration, I read this page. Does it not work?
> >
> > https://wiki.jenkins-ci.org/display/JENKINS/GitHub+OAuth+
> Plugin#GitHubOAuthPlugin-AboutGitHubAuthenticationPlugin
> >
> > On Wed, Dec 14, 2016 at 1:50 PM, Jim Apple <jb...@cloudera.com> wrote:
> >
> >> I think you're misunderstanding that plugin. It does not associate
> >> Jenkins login with gerrit login.
> >>
> >> On Wed, Dec 14, 2016 at 1:45 PM, Bharath Vissapragada
> >> <bh...@cloudera.com> wrote:
> >> > On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com>
> >> wrote:
> >> >
> >> >> That integration sounds like a great idea to me.
> >> >>
> >> >> Just to clarify the purpose: We'd like external contributors to be
> able
> >> to
> >> >> run private build+test runs during development, and not just run GVO
> >> once
> >> >> the patch has +2. The hard part is gettig the patch to +2 in the
> first
> >> >> place and we've seen instances where relying on local test runs only
> >> can be
> >> >> difficult.
> >> >>
> >> >> For example, contributors could upload a draft to gerrit and run a
> >> private
> >> >> build to fix problems before publishing the patch fir review.
> >> >>
> >> >
> >> > As per this [1] link, I think that can be configured. I haven't tried
> it
> >> > myself, but from the looks of it, it seems plausible.
> >> >
> >> > [1]
> >> > https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+
> >> Trigger#GerritTrigger-TriggerConfiguration
> >> >
> >> >
> >> >>
> >> >> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
> >> >> bharathv@cloudera.com
> >> >> > wrote:
> >> >>
> >> >> > Just wondering why we can't link Jenkins authentication with gerrit
> >> login
> >> >> > in this case instead of having two separate login credentials. That
> >> way
> >> >> we
> >> >> > can retain the audit trail of the jobs and also isolate Jenkins to
> >> only
> >> >> run
> >> >> > code thats approved (+2ed) over gerrit. With this, any new
> contributor
> >> >> > (whoever has signed up on gerrit) can have access to the jenkins
> box
> >> and
> >> >> we
> >> >> > can be sure that they only run the stuff that is approved by
> >> committers.
> >> >> > Thoughts?
> >> >> >
> >> >> > * I'm not totally sure if such an integration is possible but I
> did a
> >> >> quick
> >> >> > search and I got a feeling that shouldn't be difficult.
> >> >> >
> >> >> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
> >> >> > tbobrovytsky@cloudera.com> wrote:
> >> >> >
> >> >> > > I would be more in favor of starting with open access instead of
> >> having
> >> >> > to
> >> >> > > hand out credentials. It's both less work for us and it makes it
> >> easier
> >> >> > to
> >> >> > > contribute. If we notice that this is not working well, or gets
> >> abused,
> >> >> > we
> >> >> > > can switch to what Tim is suggesting. Also, we should be able to
> see
> >> >> who
> >> >> > is
> >> >> > > using our Jenkins by looking at Gerrit (because the patch must be
> >> >> > uploaded
> >> >> > > to Gerrit before starting a build).
> >> >> > >
> >> >> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <
> alex.behm@cloudera.com
> >> >
> >> >> > > wrote:
> >> >> > >
> >> >> > > > I'm fine with Tim's approach, but it does add some friction to
> >> >> > > > contributions.
> >> >> > > >
> >> >> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
> >> >> > tarmstrong@cloudera.com>
> >> >> > > > wrote:
> >> >> > > >
> >> >> > > > > I mean the contributor could email an email address (e.g. a
> >> mailing
> >> >> > > list)
> >> >> > > > > asking for credentials and we could email them privately.
> >> >> > > > >
> >> >> > > > > Do we know what other Apache projects do for situations like
> >> this?
> >> >> > > > >
> >> >> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <
> >> alex.behm@cloudera.com
> >> >> >
> >> >> > > > wrote:
> >> >> > > > >
> >> >> > > > > > Can you clarify the "credentials by mailing list" approach?
> >> >> > > > > >
> >> >> > > > > > If we send out the credentials on a public list, it's
> pretty
> >> >> close
> >> >> > to
> >> >> > > > > open
> >> >> > > > > > access.
> >> >> > > > > >
> >> >> > > > > > If we send out credentials to contributors privately, we
> have
> >> an
> >> >> > > > > additional
> >> >> > > > > > hurdle to contributions.
> >> >> > > > > >
> >> >> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> >> >> > > > tarmstrong@cloudera.com>
> >> >> > > > > > wrote:
> >> >> > > > > >
> >> >> > > > > > > Got it.
> >> >> > > > > > >
> >> >> > > > > > > I think I'd probably be more in favour of handing out
> login
> >> >> > > > credential
> >> >> > > > > to
> >> >> > > > > > > contributors on demand (e.g. by mailing a list) rather
> than
> >> >> > having
> >> >> > > > > open
> >> >> > > > > > > access, just so we have a clearer idea of who's using
> it. I
> >> >> don't
> >> >> > > > have
> >> >> > > > > a
> >> >> > > > > > > strong objection to the alternative.
> >> >> > > > > > >
> >> >> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
> >> >> jbapple@cloudera.com
> >> >> > >
> >> >> > > > > wrote:
> >> >> > > > > > >
> >> >> > > > > > > > > How isolated is the Jenkins instance?
> >> >> > > > > > > >
> >> >> > > > > > > > As far as I know, the workers have little access to the
> >> >> > > > coordinator.
> >> >> > > > > > See
> >> >> > > > > > > > here:
> >> >> > > > > > > >
> >> >> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> >> >> > > > > > > Master+Access+Control
> >> >> > > > > > > >
> >> >> > > > > > > > This flag is on and there are no whitelisted
> exceptions.
> >> >> > > > > > > >
> >> >> > > > > > > > > Does the jenkins user have many privileges on the VM?
> >> >> > > > > > > >
> >> >> > > > > > > > They have passwordless sudo on the worker
> >> >> > > > > > > >
> >> >> > > > > > > > > Could it simply wipe
> >> >> > > > > > > > > out the job history to destroy the trail?
> >> >> > > > > > > >
> >> >> > > > > > > > Job history is stored on the coordinator.
> >> >> > > > > > > >
> >> >> > > > > > > > > Jenkins also presumably has
> >> >> > > > > > > > > credentials to make at least some changes to gerrit -
> >> are
> >> >> > those
> >> >> > > > > > > > privileges
> >> >> > > > > > > > > restrictive enough that it couldn't cause problems
> there
> >> >> too?
> >> >> > > > > > > >
> >> >> > > > > > > > Those are stored only on the coordinator and cannot be
> >> used
> >> >> by
> >> >> > > the
> >> >> > > > > > > slaves.
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >>
> >>
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
I do not understand the proposed security benefit of using github for
user authentication over using Jenkins permissions directly. Keep in
mind that preventing non-+2ed commits from being run is an orthogonal
question - this can be configured with any user authentication method.
On Wed, Dec 14, 2016 at 1:52 PM, Bharath Vissapragada
<bh...@cloudera.com> wrote:
> That was just for configuring the gerrit trigger. Regarding auth
> integration, I read this page. Does it not work?
>
> https://wiki.jenkins-ci.org/display/JENKINS/GitHub+OAuth+Plugin#GitHubOAuthPlugin-AboutGitHubAuthenticationPlugin
>
> On Wed, Dec 14, 2016 at 1:50 PM, Jim Apple <jb...@cloudera.com> wrote:
>
>> I think you're misunderstanding that plugin. It does not associate
>> Jenkins login with gerrit login.
>>
>> On Wed, Dec 14, 2016 at 1:45 PM, Bharath Vissapragada
>> <bh...@cloudera.com> wrote:
>> > On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com>
>> wrote:
>> >
>> >> That integration sounds like a great idea to me.
>> >>
>> >> Just to clarify the purpose: We'd like external contributors to be able
>> to
>> >> run private build+test runs during development, and not just run GVO
>> once
>> >> the patch has +2. The hard part is gettig the patch to +2 in the first
>> >> place and we've seen instances where relying on local test runs only
>> can be
>> >> difficult.
>> >>
>> >> For example, contributors could upload a draft to gerrit and run a
>> private
>> >> build to fix problems before publishing the patch fir review.
>> >>
>> >
>> > As per this [1] link, I think that can be configured. I haven't tried it
>> > myself, but from the looks of it, it seems plausible.
>> >
>> > [1]
>> > https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+
>> Trigger#GerritTrigger-TriggerConfiguration
>> >
>> >
>> >>
>> >> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
>> >> bharathv@cloudera.com
>> >> > wrote:
>> >>
>> >> > Just wondering why we can't link Jenkins authentication with gerrit
>> login
>> >> > in this case instead of having two separate login credentials. That
>> way
>> >> we
>> >> > can retain the audit trail of the jobs and also isolate Jenkins to
>> only
>> >> run
>> >> > code thats approved (+2ed) over gerrit. With this, any new contributor
>> >> > (whoever has signed up on gerrit) can have access to the jenkins box
>> and
>> >> we
>> >> > can be sure that they only run the stuff that is approved by
>> committers.
>> >> > Thoughts?
>> >> >
>> >> > * I'm not totally sure if such an integration is possible but I did a
>> >> quick
>> >> > search and I got a feeling that shouldn't be difficult.
>> >> >
>> >> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
>> >> > tbobrovytsky@cloudera.com> wrote:
>> >> >
>> >> > > I would be more in favor of starting with open access instead of
>> having
>> >> > to
>> >> > > hand out credentials. It's both less work for us and it makes it
>> easier
>> >> > to
>> >> > > contribute. If we notice that this is not working well, or gets
>> abused,
>> >> > we
>> >> > > can switch to what Tim is suggesting. Also, we should be able to see
>> >> who
>> >> > is
>> >> > > using our Jenkins by looking at Gerrit (because the patch must be
>> >> > uploaded
>> >> > > to Gerrit before starting a build).
>> >> > >
>> >> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <alex.behm@cloudera.com
>> >
>> >> > > wrote:
>> >> > >
>> >> > > > I'm fine with Tim's approach, but it does add some friction to
>> >> > > > contributions.
>> >> > > >
>> >> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
>> >> > tarmstrong@cloudera.com>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > I mean the contributor could email an email address (e.g. a
>> mailing
>> >> > > list)
>> >> > > > > asking for credentials and we could email them privately.
>> >> > > > >
>> >> > > > > Do we know what other Apache projects do for situations like
>> this?
>> >> > > > >
>> >> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <
>> alex.behm@cloudera.com
>> >> >
>> >> > > > wrote:
>> >> > > > >
>> >> > > > > > Can you clarify the "credentials by mailing list" approach?
>> >> > > > > >
>> >> > > > > > If we send out the credentials on a public list, it's pretty
>> >> close
>> >> > to
>> >> > > > > open
>> >> > > > > > access.
>> >> > > > > >
>> >> > > > > > If we send out credentials to contributors privately, we have
>> an
>> >> > > > > additional
>> >> > > > > > hurdle to contributions.
>> >> > > > > >
>> >> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
>> >> > > > tarmstrong@cloudera.com>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > Got it.
>> >> > > > > > >
>> >> > > > > > > I think I'd probably be more in favour of handing out login
>> >> > > > credential
>> >> > > > > to
>> >> > > > > > > contributors on demand (e.g. by mailing a list) rather than
>> >> > having
>> >> > > > > open
>> >> > > > > > > access, just so we have a clearer idea of who's using it. I
>> >> don't
>> >> > > > have
>> >> > > > > a
>> >> > > > > > > strong objection to the alternative.
>> >> > > > > > >
>> >> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
>> >> jbapple@cloudera.com
>> >> > >
>> >> > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > > How isolated is the Jenkins instance?
>> >> > > > > > > >
>> >> > > > > > > > As far as I know, the workers have little access to the
>> >> > > > coordinator.
>> >> > > > > > See
>> >> > > > > > > > here:
>> >> > > > > > > >
>> >> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
>> >> > > > > > > Master+Access+Control
>> >> > > > > > > >
>> >> > > > > > > > This flag is on and there are no whitelisted exceptions.
>> >> > > > > > > >
>> >> > > > > > > > > Does the jenkins user have many privileges on the VM?
>> >> > > > > > > >
>> >> > > > > > > > They have passwordless sudo on the worker
>> >> > > > > > > >
>> >> > > > > > > > > Could it simply wipe
>> >> > > > > > > > > out the job history to destroy the trail?
>> >> > > > > > > >
>> >> > > > > > > > Job history is stored on the coordinator.
>> >> > > > > > > >
>> >> > > > > > > > > Jenkins also presumably has
>> >> > > > > > > > > credentials to make at least some changes to gerrit -
>> are
>> >> > those
>> >> > > > > > > > privileges
>> >> > > > > > > > > restrictive enough that it couldn't cause problems there
>> >> too?
>> >> > > > > > > >
>> >> > > > > > > > Those are stored only on the coordinator and cannot be
>> used
>> >> by
>> >> > > the
>> >> > > > > > > slaves.
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
Re: Jenkins jobs to verify patches
Posted by Bharath Vissapragada <bh...@cloudera.com>.
That was just for configuring the gerrit trigger. Regarding auth
integration, I read this page. Does it not work?
https://wiki.jenkins-ci.org/display/JENKINS/GitHub+OAuth+Plugin#GitHubOAuthPlugin-AboutGitHubAuthenticationPlugin
On Wed, Dec 14, 2016 at 1:50 PM, Jim Apple <jb...@cloudera.com> wrote:
> I think you're misunderstanding that plugin. It does not associate
> Jenkins login with gerrit login.
>
> On Wed, Dec 14, 2016 at 1:45 PM, Bharath Vissapragada
> <bh...@cloudera.com> wrote:
> > On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com>
> wrote:
> >
> >> That integration sounds like a great idea to me.
> >>
> >> Just to clarify the purpose: We'd like external contributors to be able
> to
> >> run private build+test runs during development, and not just run GVO
> once
> >> the patch has +2. The hard part is gettig the patch to +2 in the first
> >> place and we've seen instances where relying on local test runs only
> can be
> >> difficult.
> >>
> >> For example, contributors could upload a draft to gerrit and run a
> private
> >> build to fix problems before publishing the patch fir review.
> >>
> >
> > As per this [1] link, I think that can be configured. I haven't tried it
> > myself, but from the looks of it, it seems plausible.
> >
> > [1]
> > https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+
> Trigger#GerritTrigger-TriggerConfiguration
> >
> >
> >>
> >> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
> >> bharathv@cloudera.com
> >> > wrote:
> >>
> >> > Just wondering why we can't link Jenkins authentication with gerrit
> login
> >> > in this case instead of having two separate login credentials. That
> way
> >> we
> >> > can retain the audit trail of the jobs and also isolate Jenkins to
> only
> >> run
> >> > code thats approved (+2ed) over gerrit. With this, any new contributor
> >> > (whoever has signed up on gerrit) can have access to the jenkins box
> and
> >> we
> >> > can be sure that they only run the stuff that is approved by
> committers.
> >> > Thoughts?
> >> >
> >> > * I'm not totally sure if such an integration is possible but I did a
> >> quick
> >> > search and I got a feeling that shouldn't be difficult.
> >> >
> >> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
> >> > tbobrovytsky@cloudera.com> wrote:
> >> >
> >> > > I would be more in favor of starting with open access instead of
> having
> >> > to
> >> > > hand out credentials. It's both less work for us and it makes it
> easier
> >> > to
> >> > > contribute. If we notice that this is not working well, or gets
> abused,
> >> > we
> >> > > can switch to what Tim is suggesting. Also, we should be able to see
> >> who
> >> > is
> >> > > using our Jenkins by looking at Gerrit (because the patch must be
> >> > uploaded
> >> > > to Gerrit before starting a build).
> >> > >
> >> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <alex.behm@cloudera.com
> >
> >> > > wrote:
> >> > >
> >> > > > I'm fine with Tim's approach, but it does add some friction to
> >> > > > contributions.
> >> > > >
> >> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
> >> > tarmstrong@cloudera.com>
> >> > > > wrote:
> >> > > >
> >> > > > > I mean the contributor could email an email address (e.g. a
> mailing
> >> > > list)
> >> > > > > asking for credentials and we could email them privately.
> >> > > > >
> >> > > > > Do we know what other Apache projects do for situations like
> this?
> >> > > > >
> >> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <
> alex.behm@cloudera.com
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Can you clarify the "credentials by mailing list" approach?
> >> > > > > >
> >> > > > > > If we send out the credentials on a public list, it's pretty
> >> close
> >> > to
> >> > > > > open
> >> > > > > > access.
> >> > > > > >
> >> > > > > > If we send out credentials to contributors privately, we have
> an
> >> > > > > additional
> >> > > > > > hurdle to contributions.
> >> > > > > >
> >> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> >> > > > tarmstrong@cloudera.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Got it.
> >> > > > > > >
> >> > > > > > > I think I'd probably be more in favour of handing out login
> >> > > > credential
> >> > > > > to
> >> > > > > > > contributors on demand (e.g. by mailing a list) rather than
> >> > having
> >> > > > > open
> >> > > > > > > access, just so we have a clearer idea of who's using it. I
> >> don't
> >> > > > have
> >> > > > > a
> >> > > > > > > strong objection to the alternative.
> >> > > > > > >
> >> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
> >> jbapple@cloudera.com
> >> > >
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > > How isolated is the Jenkins instance?
> >> > > > > > > >
> >> > > > > > > > As far as I know, the workers have little access to the
> >> > > > coordinator.
> >> > > > > > See
> >> > > > > > > > here:
> >> > > > > > > >
> >> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> >> > > > > > > Master+Access+Control
> >> > > > > > > >
> >> > > > > > > > This flag is on and there are no whitelisted exceptions.
> >> > > > > > > >
> >> > > > > > > > > Does the jenkins user have many privileges on the VM?
> >> > > > > > > >
> >> > > > > > > > They have passwordless sudo on the worker
> >> > > > > > > >
> >> > > > > > > > > Could it simply wipe
> >> > > > > > > > > out the job history to destroy the trail?
> >> > > > > > > >
> >> > > > > > > > Job history is stored on the coordinator.
> >> > > > > > > >
> >> > > > > > > > > Jenkins also presumably has
> >> > > > > > > > > credentials to make at least some changes to gerrit -
> are
> >> > those
> >> > > > > > > > privileges
> >> > > > > > > > > restrictive enough that it couldn't cause problems there
> >> too?
> >> > > > > > > >
> >> > > > > > > > Those are stored only on the coordinator and cannot be
> used
> >> by
> >> > > the
> >> > > > > > > slaves.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
I think you're misunderstanding that plugin. It does not associate
Jenkins login with gerrit login.
On Wed, Dec 14, 2016 at 1:45 PM, Bharath Vissapragada
<bh...@cloudera.com> wrote:
> On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com> wrote:
>
>> That integration sounds like a great idea to me.
>>
>> Just to clarify the purpose: We'd like external contributors to be able to
>> run private build+test runs during development, and not just run GVO once
>> the patch has +2. The hard part is gettig the patch to +2 in the first
>> place and we've seen instances where relying on local test runs only can be
>> difficult.
>>
>> For example, contributors could upload a draft to gerrit and run a private
>> build to fix problems before publishing the patch fir review.
>>
>
> As per this [1] link, I think that can be configured. I haven't tried it
> myself, but from the looks of it, it seems plausible.
>
> [1]
> https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-TriggerConfiguration
>
>
>>
>> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
>> bharathv@cloudera.com
>> > wrote:
>>
>> > Just wondering why we can't link Jenkins authentication with gerrit login
>> > in this case instead of having two separate login credentials. That way
>> we
>> > can retain the audit trail of the jobs and also isolate Jenkins to only
>> run
>> > code thats approved (+2ed) over gerrit. With this, any new contributor
>> > (whoever has signed up on gerrit) can have access to the jenkins box and
>> we
>> > can be sure that they only run the stuff that is approved by committers.
>> > Thoughts?
>> >
>> > * I'm not totally sure if such an integration is possible but I did a
>> quick
>> > search and I got a feeling that shouldn't be difficult.
>> >
>> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
>> > tbobrovytsky@cloudera.com> wrote:
>> >
>> > > I would be more in favor of starting with open access instead of having
>> > to
>> > > hand out credentials. It's both less work for us and it makes it easier
>> > to
>> > > contribute. If we notice that this is not working well, or gets abused,
>> > we
>> > > can switch to what Tim is suggesting. Also, we should be able to see
>> who
>> > is
>> > > using our Jenkins by looking at Gerrit (because the patch must be
>> > uploaded
>> > > to Gerrit before starting a build).
>> > >
>> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <al...@cloudera.com>
>> > > wrote:
>> > >
>> > > > I'm fine with Tim's approach, but it does add some friction to
>> > > > contributions.
>> > > >
>> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
>> > tarmstrong@cloudera.com>
>> > > > wrote:
>> > > >
>> > > > > I mean the contributor could email an email address (e.g. a mailing
>> > > list)
>> > > > > asking for credentials and we could email them privately.
>> > > > >
>> > > > > Do we know what other Apache projects do for situations like this?
>> > > > >
>> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <alex.behm@cloudera.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Can you clarify the "credentials by mailing list" approach?
>> > > > > >
>> > > > > > If we send out the credentials on a public list, it's pretty
>> close
>> > to
>> > > > > open
>> > > > > > access.
>> > > > > >
>> > > > > > If we send out credentials to contributors privately, we have an
>> > > > > additional
>> > > > > > hurdle to contributions.
>> > > > > >
>> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
>> > > > tarmstrong@cloudera.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Got it.
>> > > > > > >
>> > > > > > > I think I'd probably be more in favour of handing out login
>> > > > credential
>> > > > > to
>> > > > > > > contributors on demand (e.g. by mailing a list) rather than
>> > having
>> > > > > open
>> > > > > > > access, just so we have a clearer idea of who's using it. I
>> don't
>> > > > have
>> > > > > a
>> > > > > > > strong objection to the alternative.
>> > > > > > >
>> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
>> jbapple@cloudera.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > > How isolated is the Jenkins instance?
>> > > > > > > >
>> > > > > > > > As far as I know, the workers have little access to the
>> > > > coordinator.
>> > > > > > See
>> > > > > > > > here:
>> > > > > > > >
>> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
>> > > > > > > Master+Access+Control
>> > > > > > > >
>> > > > > > > > This flag is on and there are no whitelisted exceptions.
>> > > > > > > >
>> > > > > > > > > Does the jenkins user have many privileges on the VM?
>> > > > > > > >
>> > > > > > > > They have passwordless sudo on the worker
>> > > > > > > >
>> > > > > > > > > Could it simply wipe
>> > > > > > > > > out the job history to destroy the trail?
>> > > > > > > >
>> > > > > > > > Job history is stored on the coordinator.
>> > > > > > > >
>> > > > > > > > > Jenkins also presumably has
>> > > > > > > > > credentials to make at least some changes to gerrit - are
>> > those
>> > > > > > > > privileges
>> > > > > > > > > restrictive enough that it couldn't cause problems there
>> too?
>> > > > > > > >
>> > > > > > > > Those are stored only on the coordinator and cannot be used
>> by
>> > > the
>> > > > > > > slaves.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
Re: Jenkins jobs to verify patches
Posted by Bharath Vissapragada <bh...@cloudera.com>.
On Wed, Dec 14, 2016 at 1:38 PM, Alex Behm <al...@cloudera.com> wrote:
> That integration sounds like a great idea to me.
>
> Just to clarify the purpose: We'd like external contributors to be able to
> run private build+test runs during development, and not just run GVO once
> the patch has +2. The hard part is gettig the patch to +2 in the first
> place and we've seen instances where relying on local test runs only can be
> difficult.
>
> For example, contributors could upload a draft to gerrit and run a private
> build to fix problems before publishing the patch fir review.
>
As per this [1] link, I think that can be configured. I haven't tried it
myself, but from the looks of it, it seems plausible.
[1]
https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-TriggerConfiguration
>
> On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <
> bharathv@cloudera.com
> > wrote:
>
> > Just wondering why we can't link Jenkins authentication with gerrit login
> > in this case instead of having two separate login credentials. That way
> we
> > can retain the audit trail of the jobs and also isolate Jenkins to only
> run
> > code thats approved (+2ed) over gerrit. With this, any new contributor
> > (whoever has signed up on gerrit) can have access to the jenkins box and
> we
> > can be sure that they only run the stuff that is approved by committers.
> > Thoughts?
> >
> > * I'm not totally sure if such an integration is possible but I did a
> quick
> > search and I got a feeling that shouldn't be difficult.
> >
> > On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
> > tbobrovytsky@cloudera.com> wrote:
> >
> > > I would be more in favor of starting with open access instead of having
> > to
> > > hand out credentials. It's both less work for us and it makes it easier
> > to
> > > contribute. If we notice that this is not working well, or gets abused,
> > we
> > > can switch to what Tim is suggesting. Also, we should be able to see
> who
> > is
> > > using our Jenkins by looking at Gerrit (because the patch must be
> > uploaded
> > > to Gerrit before starting a build).
> > >
> > > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <al...@cloudera.com>
> > > wrote:
> > >
> > > > I'm fine with Tim's approach, but it does add some friction to
> > > > contributions.
> > > >
> > > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
> > tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > > > I mean the contributor could email an email address (e.g. a mailing
> > > list)
> > > > > asking for credentials and we could email them privately.
> > > > >
> > > > > Do we know what other Apache projects do for situations like this?
> > > > >
> > > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <alex.behm@cloudera.com
> >
> > > > wrote:
> > > > >
> > > > > > Can you clarify the "credentials by mailing list" approach?
> > > > > >
> > > > > > If we send out the credentials on a public list, it's pretty
> close
> > to
> > > > > open
> > > > > > access.
> > > > > >
> > > > > > If we send out credentials to contributors privately, we have an
> > > > > additional
> > > > > > hurdle to contributions.
> > > > > >
> > > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> > > > tarmstrong@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Got it.
> > > > > > >
> > > > > > > I think I'd probably be more in favour of handing out login
> > > > credential
> > > > > to
> > > > > > > contributors on demand (e.g. by mailing a list) rather than
> > having
> > > > > open
> > > > > > > access, just so we have a clearer idea of who's using it. I
> don't
> > > > have
> > > > > a
> > > > > > > strong objection to the alternative.
> > > > > > >
> > > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <
> jbapple@cloudera.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > > How isolated is the Jenkins instance?
> > > > > > > >
> > > > > > > > As far as I know, the workers have little access to the
> > > > coordinator.
> > > > > > See
> > > > > > > > here:
> > > > > > > >
> > > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > > > > > Master+Access+Control
> > > > > > > >
> > > > > > > > This flag is on and there are no whitelisted exceptions.
> > > > > > > >
> > > > > > > > > Does the jenkins user have many privileges on the VM?
> > > > > > > >
> > > > > > > > They have passwordless sudo on the worker
> > > > > > > >
> > > > > > > > > Could it simply wipe
> > > > > > > > > out the job history to destroy the trail?
> > > > > > > >
> > > > > > > > Job history is stored on the coordinator.
> > > > > > > >
> > > > > > > > > Jenkins also presumably has
> > > > > > > > > credentials to make at least some changes to gerrit - are
> > those
> > > > > > > > privileges
> > > > > > > > > restrictive enough that it couldn't cause problems there
> too?
> > > > > > > >
> > > > > > > > Those are stored only on the coordinator and cannot be used
> by
> > > the
> > > > > > > slaves.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Alex Behm <al...@cloudera.com>.
That integration sounds like a great idea to me.
Just to clarify the purpose: We'd like external contributors to be able to
run private build+test runs during development, and not just run GVO once
the patch has +2. The hard part is gettig the patch to +2 in the first
place and we've seen instances where relying on local test runs only can be
difficult.
For example, contributors could upload a draft to gerrit and run a private
build to fix problems before publishing the patch fir review.
On Wed, Dec 14, 2016 at 1:33 PM, Bharath Vissapragada <bharathv@cloudera.com
> wrote:
> Just wondering why we can't link Jenkins authentication with gerrit login
> in this case instead of having two separate login credentials. That way we
> can retain the audit trail of the jobs and also isolate Jenkins to only run
> code thats approved (+2ed) over gerrit. With this, any new contributor
> (whoever has signed up on gerrit) can have access to the jenkins box and we
> can be sure that they only run the stuff that is approved by committers.
> Thoughts?
>
> * I'm not totally sure if such an integration is possible but I did a quick
> search and I got a feeling that shouldn't be difficult.
>
> On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
> tbobrovytsky@cloudera.com> wrote:
>
> > I would be more in favor of starting with open access instead of having
> to
> > hand out credentials. It's both less work for us and it makes it easier
> to
> > contribute. If we notice that this is not working well, or gets abused,
> we
> > can switch to what Tim is suggesting. Also, we should be able to see who
> is
> > using our Jenkins by looking at Gerrit (because the patch must be
> uploaded
> > to Gerrit before starting a build).
> >
> > On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <al...@cloudera.com>
> > wrote:
> >
> > > I'm fine with Tim's approach, but it does add some friction to
> > > contributions.
> > >
> > > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > > > I mean the contributor could email an email address (e.g. a mailing
> > list)
> > > > asking for credentials and we could email them privately.
> > > >
> > > > Do we know what other Apache projects do for situations like this?
> > > >
> > > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com>
> > > wrote:
> > > >
> > > > > Can you clarify the "credentials by mailing list" approach?
> > > > >
> > > > > If we send out the credentials on a public list, it's pretty close
> to
> > > > open
> > > > > access.
> > > > >
> > > > > If we send out credentials to contributors privately, we have an
> > > > additional
> > > > > hurdle to contributions.
> > > > >
> > > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> > > tarmstrong@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Got it.
> > > > > >
> > > > > > I think I'd probably be more in favour of handing out login
> > > credential
> > > > to
> > > > > > contributors on demand (e.g. by mailing a list) rather than
> having
> > > > open
> > > > > > access, just so we have a clearer idea of who's using it. I don't
> > > have
> > > > a
> > > > > > strong objection to the alternative.
> > > > > >
> > > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jbapple@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > > How isolated is the Jenkins instance?
> > > > > > >
> > > > > > > As far as I know, the workers have little access to the
> > > coordinator.
> > > > > See
> > > > > > > here:
> > > > > > >
> > > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > > > > Master+Access+Control
> > > > > > >
> > > > > > > This flag is on and there are no whitelisted exceptions.
> > > > > > >
> > > > > > > > Does the jenkins user have many privileges on the VM?
> > > > > > >
> > > > > > > They have passwordless sudo on the worker
> > > > > > >
> > > > > > > > Could it simply wipe
> > > > > > > > out the job history to destroy the trail?
> > > > > > >
> > > > > > > Job history is stored on the coordinator.
> > > > > > >
> > > > > > > > Jenkins also presumably has
> > > > > > > > credentials to make at least some changes to gerrit - are
> those
> > > > > > > privileges
> > > > > > > > restrictive enough that it couldn't cause problems there too?
> > > > > > >
> > > > > > > Those are stored only on the coordinator and cannot be used by
> > the
> > > > > > slaves.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Just wondering why we can't link Jenkins authentication with gerrit login
in this case instead of having two separate login credentials. That way we
can retain the audit trail of the jobs and also isolate Jenkins to only run
code thats approved (+2ed) over gerrit. With this, any new contributor
(whoever has signed up on gerrit) can have access to the jenkins box and we
can be sure that they only run the stuff that is approved by committers.
Thoughts?
* I'm not totally sure if such an integration is possible but I did a quick
search and I got a feeling that shouldn't be difficult.
On Wed, Dec 14, 2016 at 1:18 PM, Taras Bobrovytsky <
tbobrovytsky@cloudera.com> wrote:
> I would be more in favor of starting with open access instead of having to
> hand out credentials. It's both less work for us and it makes it easier to
> contribute. If we notice that this is not working well, or gets abused, we
> can switch to what Tim is suggesting. Also, we should be able to see who is
> using our Jenkins by looking at Gerrit (because the patch must be uploaded
> to Gerrit before starting a build).
>
> On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <al...@cloudera.com>
> wrote:
>
> > I'm fine with Tim's approach, but it does add some friction to
> > contributions.
> >
> > On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > I mean the contributor could email an email address (e.g. a mailing
> list)
> > > asking for credentials and we could email them privately.
> > >
> > > Do we know what other Apache projects do for situations like this?
> > >
> > > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com>
> > wrote:
> > >
> > > > Can you clarify the "credentials by mailing list" approach?
> > > >
> > > > If we send out the credentials on a public list, it's pretty close to
> > > open
> > > > access.
> > > >
> > > > If we send out credentials to contributors privately, we have an
> > > additional
> > > > hurdle to contributions.
> > > >
> > > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> > tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > > > Got it.
> > > > >
> > > > > I think I'd probably be more in favour of handing out login
> > credential
> > > to
> > > > > contributors on demand (e.g. by mailing a list) rather than having
> > > open
> > > > > access, just so we have a clearer idea of who's using it. I don't
> > have
> > > a
> > > > > strong objection to the alternative.
> > > > >
> > > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > > How isolated is the Jenkins instance?
> > > > > >
> > > > > > As far as I know, the workers have little access to the
> > coordinator.
> > > > See
> > > > > > here:
> > > > > >
> > > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > > > Master+Access+Control
> > > > > >
> > > > > > This flag is on and there are no whitelisted exceptions.
> > > > > >
> > > > > > > Does the jenkins user have many privileges on the VM?
> > > > > >
> > > > > > They have passwordless sudo on the worker
> > > > > >
> > > > > > > Could it simply wipe
> > > > > > > out the job history to destroy the trail?
> > > > > >
> > > > > > Job history is stored on the coordinator.
> > > > > >
> > > > > > > Jenkins also presumably has
> > > > > > > credentials to make at least some changes to gerrit - are those
> > > > > > privileges
> > > > > > > restrictive enough that it couldn't cause problems there too?
> > > > > >
> > > > > > Those are stored only on the coordinator and cannot be used by
> the
> > > > > slaves.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Taras Bobrovytsky <tb...@cloudera.com>.
I would be more in favor of starting with open access instead of having to
hand out credentials. It's both less work for us and it makes it easier to
contribute. If we notice that this is not working well, or gets abused, we
can switch to what Tim is suggesting. Also, we should be able to see who is
using our Jenkins by looking at Gerrit (because the patch must be uploaded
to Gerrit before starting a build).
On Wed, Dec 14, 2016 at 10:57 AM, Alex Behm <al...@cloudera.com> wrote:
> I'm fine with Tim's approach, but it does add some friction to
> contributions.
>
> On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > I mean the contributor could email an email address (e.g. a mailing list)
> > asking for credentials and we could email them privately.
> >
> > Do we know what other Apache projects do for situations like this?
> >
> > On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com>
> wrote:
> >
> > > Can you clarify the "credentials by mailing list" approach?
> > >
> > > If we send out the credentials on a public list, it's pretty close to
> > open
> > > access.
> > >
> > > If we send out credentials to contributors privately, we have an
> > additional
> > > hurdle to contributions.
> > >
> > > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > > > Got it.
> > > >
> > > > I think I'd probably be more in favour of handing out login
> credential
> > to
> > > > contributors on demand (e.g. by mailing a list) rather than having
> > open
> > > > access, just so we have a clearer idea of who's using it. I don't
> have
> > a
> > > > strong objection to the alternative.
> > > >
> > > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
> > wrote:
> > > >
> > > > > > How isolated is the Jenkins instance?
> > > > >
> > > > > As far as I know, the workers have little access to the
> coordinator.
> > > See
> > > > > here:
> > > > >
> > > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > > Master+Access+Control
> > > > >
> > > > > This flag is on and there are no whitelisted exceptions.
> > > > >
> > > > > > Does the jenkins user have many privileges on the VM?
> > > > >
> > > > > They have passwordless sudo on the worker
> > > > >
> > > > > > Could it simply wipe
> > > > > > out the job history to destroy the trail?
> > > > >
> > > > > Job history is stored on the coordinator.
> > > > >
> > > > > > Jenkins also presumably has
> > > > > > credentials to make at least some changes to gerrit - are those
> > > > > privileges
> > > > > > restrictive enough that it couldn't cause problems there too?
> > > > >
> > > > > Those are stored only on the coordinator and cannot be used by the
> > > > slaves.
> > > > >
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Alex Behm <al...@cloudera.com>.
I'm fine with Tim's approach, but it does add some friction to
contributions.
On Wed, Dec 14, 2016 at 9:57 AM, Tim Armstrong <ta...@cloudera.com>
wrote:
> I mean the contributor could email an email address (e.g. a mailing list)
> asking for credentials and we could email them privately.
>
> Do we know what other Apache projects do for situations like this?
>
> On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com> wrote:
>
> > Can you clarify the "credentials by mailing list" approach?
> >
> > If we send out the credentials on a public list, it's pretty close to
> open
> > access.
> >
> > If we send out credentials to contributors privately, we have an
> additional
> > hurdle to contributions.
> >
> > On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > Got it.
> > >
> > > I think I'd probably be more in favour of handing out login credential
> to
> > > contributors on demand (e.g. by mailing a list) rather than having
> open
> > > access, just so we have a clearer idea of who's using it. I don't have
> a
> > > strong objection to the alternative.
> > >
> > > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com>
> wrote:
> > >
> > > > > How isolated is the Jenkins instance?
> > > >
> > > > As far as I know, the workers have little access to the coordinator.
> > See
> > > > here:
> > > >
> > > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > > Master+Access+Control
> > > >
> > > > This flag is on and there are no whitelisted exceptions.
> > > >
> > > > > Does the jenkins user have many privileges on the VM?
> > > >
> > > > They have passwordless sudo on the worker
> > > >
> > > > > Could it simply wipe
> > > > > out the job history to destroy the trail?
> > > >
> > > > Job history is stored on the coordinator.
> > > >
> > > > > Jenkins also presumably has
> > > > > credentials to make at least some changes to gerrit - are those
> > > > privileges
> > > > > restrictive enough that it couldn't cause problems there too?
> > > >
> > > > Those are stored only on the coordinator and cannot be used by the
> > > slaves.
> > > >
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Tim Armstrong <ta...@cloudera.com>.
I mean the contributor could email an email address (e.g. a mailing list)
asking for credentials and we could email them privately.
Do we know what other Apache projects do for situations like this?
On Wed, Dec 14, 2016 at 9:18 AM, Alex Behm <al...@cloudera.com> wrote:
> Can you clarify the "credentials by mailing list" approach?
>
> If we send out the credentials on a public list, it's pretty close to open
> access.
>
> If we send out credentials to contributors privately, we have an additional
> hurdle to contributions.
>
> On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > Got it.
> >
> > I think I'd probably be more in favour of handing out login credential to
> > contributors on demand (e.g. by mailing a list) rather than having open
> > access, just so we have a clearer idea of who's using it. I don't have a
> > strong objection to the alternative.
> >
> > On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com> wrote:
> >
> > > > How isolated is the Jenkins instance?
> > >
> > > As far as I know, the workers have little access to the coordinator.
> See
> > > here:
> > >
> > > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> > Master+Access+Control
> > >
> > > This flag is on and there are no whitelisted exceptions.
> > >
> > > > Does the jenkins user have many privileges on the VM?
> > >
> > > They have passwordless sudo on the worker
> > >
> > > > Could it simply wipe
> > > > out the job history to destroy the trail?
> > >
> > > Job history is stored on the coordinator.
> > >
> > > > Jenkins also presumably has
> > > > credentials to make at least some changes to gerrit - are those
> > > privileges
> > > > restrictive enough that it couldn't cause problems there too?
> > >
> > > Those are stored only on the coordinator and cannot be used by the
> > slaves.
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Alex Behm <al...@cloudera.com>.
Can you clarify the "credentials by mailing list" approach?
If we send out the credentials on a public list, it's pretty close to open
access.
If we send out credentials to contributors privately, we have an additional
hurdle to contributions.
On Wed, Dec 14, 2016 at 9:12 AM, Tim Armstrong <ta...@cloudera.com>
wrote:
> Got it.
>
> I think I'd probably be more in favour of handing out login credential to
> contributors on demand (e.g. by mailing a list) rather than having open
> access, just so we have a clearer idea of who's using it. I don't have a
> strong objection to the alternative.
>
> On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com> wrote:
>
> > > How isolated is the Jenkins instance?
> >
> > As far as I know, the workers have little access to the coordinator. See
> > here:
> >
> > https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+
> Master+Access+Control
> >
> > This flag is on and there are no whitelisted exceptions.
> >
> > > Does the jenkins user have many privileges on the VM?
> >
> > They have passwordless sudo on the worker
> >
> > > Could it simply wipe
> > > out the job history to destroy the trail?
> >
> > Job history is stored on the coordinator.
> >
> > > Jenkins also presumably has
> > > credentials to make at least some changes to gerrit - are those
> > privileges
> > > restrictive enough that it couldn't cause problems there too?
> >
> > Those are stored only on the coordinator and cannot be used by the
> slaves.
> >
>
Re: Jenkins jobs to verify patches
Posted by Tim Armstrong <ta...@cloudera.com>.
Got it.
I think I'd probably be more in favour of handing out login credential to
contributors on demand (e.g. by mailing a list) rather than having open
access, just so we have a clearer idea of who's using it. I don't have a
strong objection to the alternative.
On Wed, Dec 14, 2016 at 8:52 AM, Jim Apple <jb...@cloudera.com> wrote:
> > How isolated is the Jenkins instance?
>
> As far as I know, the workers have little access to the coordinator. See
> here:
>
> https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+Master+Access+Control
>
> This flag is on and there are no whitelisted exceptions.
>
> > Does the jenkins user have many privileges on the VM?
>
> They have passwordless sudo on the worker
>
> > Could it simply wipe
> > out the job history to destroy the trail?
>
> Job history is stored on the coordinator.
>
> > Jenkins also presumably has
> > credentials to make at least some changes to gerrit - are those
> privileges
> > restrictive enough that it couldn't cause problems there too?
>
> Those are stored only on the coordinator and cannot be used by the slaves.
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
> How isolated is the Jenkins instance?
As far as I know, the workers have little access to the coordinator. See here:
https://wiki.jenkins-ci.org/display/JENKINS/Slave+To+Master+Access+Control
This flag is on and there are no whitelisted exceptions.
> Does the jenkins user have many privileges on the VM?
They have passwordless sudo on the worker
> Could it simply wipe
> out the job history to destroy the trail?
Job history is stored on the coordinator.
> Jenkins also presumably has
> credentials to make at least some changes to gerrit - are those privileges
> restrictive enough that it couldn't cause problems there too?
Those are stored only on the coordinator and cannot be used by the slaves.
Re: Jenkins jobs to verify patches
Posted by Tim Armstrong <ta...@cloudera.com>.
Thanks for all your hard work on this Jim. This has been a major pain point
for external contributors.
I agree with the goal of minimising friction, but I'm just trying to think
through the potential consequences of leaving the job open.
How isolated is the Jenkins instance? We'd be allowing people to execute
arbitrary code on that box as far as I understand it, so we'd want to be
very confident that it didn't have any consequences beyond compromising the
VM.
Does the jenkins user have many privileges on the VM? Could it simply wipe
out the job history to destroy the trail? Jenkins also presumably has
credentials to make at least some changes to gerrit - are those privileges
restrictive enough that it couldn't cause problems there too?
- Tim
On Tue, Dec 13, 2016 at 8:33 PM, Jim Apple <jb...@cloudera.com> wrote:
> I'm on board with all of this so far.
>
> On Tue, Dec 13, 2016 at 5:30 PM, Marcel Kornacker <ma...@cloudera.com>
> wrote:
> > Regarding suggestion 1: let's make that 01/09 or later to guarantee a
> > smoother transition.
> >
> > On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
> >> Before Impala joined the ASF, code reviews had to go through a
> >> "verification" step in which a Jenkins jobs downloaded the patch, ran
> >> all tests and replied back to Jenkins that everything was OK. That
> >> Jenkins job ran on Cloudera infrastructure and could not be accessed,
> >> even in a read-only way, by people outside of Cloudera.
> >>
> >> To follow the Apache way, I am laboring to replace that with a Jenkins
> >> server that can be used by any authorized person and read by any
> >> person. It is at http://jenkins.impala.io:8080. It is able to verify
> >> patches just like the old Jenkins machine. A few remaining questions:
> >>
> >> 1. What should the prerequisites be to turning off the Cloudera-only
> >> Jenkins verification path?
> >>
> >> 2. Who should be able to run jobs on jenkins.impala.io? Some
> >> possibilities: Committers only, anyone who asks, PMC members only,
> >> contributors who ask after submitting 5 patches. Higher bars lead to
> >> less likelihood of abuse, lower ones to easier contributions from
> >> newbies.
> >>
> >> My proposal is this:
> >>
> >> 1. We should turn off the CLoudera-only Jenkins verification path
> >> January 2. jenkins.impala.io is in pretty good shape, and we can make
> >> further improvements as needed. For instance, it took me five minutes
> >> just now to cut down on the not-so-interesting debug output from the
> >> main verification job,
> >> http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> >>
> >> 2. Everyone with five patches can request a login.
> >>
> >> I'm not married to these ideas, but I wanted to provide a jumping-off
> >> point for discussion.
>
Re: Jenkins jobs to verify patches
Posted by Jim Apple <jb...@cloudera.com>.
I'm on board with all of this so far.
On Tue, Dec 13, 2016 at 5:30 PM, Marcel Kornacker <ma...@cloudera.com> wrote:
> Regarding suggestion 1: let's make that 01/09 or later to guarantee a
> smoother transition.
>
> On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
>> Before Impala joined the ASF, code reviews had to go through a
>> "verification" step in which a Jenkins jobs downloaded the patch, ran
>> all tests and replied back to Jenkins that everything was OK. That
>> Jenkins job ran on Cloudera infrastructure and could not be accessed,
>> even in a read-only way, by people outside of Cloudera.
>>
>> To follow the Apache way, I am laboring to replace that with a Jenkins
>> server that can be used by any authorized person and read by any
>> person. It is at http://jenkins.impala.io:8080. It is able to verify
>> patches just like the old Jenkins machine. A few remaining questions:
>>
>> 1. What should the prerequisites be to turning off the Cloudera-only
>> Jenkins verification path?
>>
>> 2. Who should be able to run jobs on jenkins.impala.io? Some
>> possibilities: Committers only, anyone who asks, PMC members only,
>> contributors who ask after submitting 5 patches. Higher bars lead to
>> less likelihood of abuse, lower ones to easier contributions from
>> newbies.
>>
>> My proposal is this:
>>
>> 1. We should turn off the CLoudera-only Jenkins verification path
>> January 2. jenkins.impala.io is in pretty good shape, and we can make
>> further improvements as needed. For instance, it took me five minutes
>> just now to cut down on the not-so-interesting debug output from the
>> main verification job,
>> http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
>>
>> 2. Everyone with five patches can request a login.
>>
>> I'm not married to these ideas, but I wanted to provide a jumping-off
>> point for discussion.
Re: Jenkins jobs to verify patches
Posted by Marcel Kornacker <ma...@cloudera.com>.
Regarding suggestion 1: let's make that 01/09 or later to guarantee a
smoother transition.
On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
> Before Impala joined the ASF, code reviews had to go through a
> "verification" step in which a Jenkins jobs downloaded the patch, ran
> all tests and replied back to Jenkins that everything was OK. That
> Jenkins job ran on Cloudera infrastructure and could not be accessed,
> even in a read-only way, by people outside of Cloudera.
>
> To follow the Apache way, I am laboring to replace that with a Jenkins
> server that can be used by any authorized person and read by any
> person. It is at http://jenkins.impala.io:8080. It is able to verify
> patches just like the old Jenkins machine. A few remaining questions:
>
> 1. What should the prerequisites be to turning off the Cloudera-only
> Jenkins verification path?
>
> 2. Who should be able to run jobs on jenkins.impala.io? Some
> possibilities: Committers only, anyone who asks, PMC members only,
> contributors who ask after submitting 5 patches. Higher bars lead to
> less likelihood of abuse, lower ones to easier contributions from
> newbies.
>
> My proposal is this:
>
> 1. We should turn off the CLoudera-only Jenkins verification path
> January 2. jenkins.impala.io is in pretty good shape, and we can make
> further improvements as needed. For instance, it took me five minutes
> just now to cut down on the not-so-interesting debug output from the
> main verification job,
> http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
>
> 2. Everyone with five patches can request a login.
>
> I'm not married to these ideas, but I wanted to provide a jumping-off
> point for discussion.
Re: Jenkins jobs to verify patches
Posted by Alex Behm <al...@cloudera.com>.
Yes, Either no login or a trivial login like anon/anon.
On Tue, Dec 13, 2016 at 5:27 PM, Taras Bobrovytsky <
tbobrovytsky@cloudera.com> wrote:
> I generally agree with Alex's thoughts on this. Just to clarify, the
> suggestion is that the anonymous private build-and-test job can be started
> without having to log in to Jenkins?
>
> On Tue, Dec 13, 2016 at 4:55 PM, Alex Behm <al...@cloudera.com> wrote:
>
> > The way I see it we have two competing interests that need to be
> > reconciled:
> > 1. We want to encourage contributions by making it as easy as possible to
> > do private test runs. This speaks in favor of allowing any contributor to
> > run on jenkins.impala.io.
> > 2. Cloudera pays the bill for the EC2 Jenkins workers, so we do need to
> > protect Cloudera against abuse.
> >
> > I am strongly in favor of (1) and I think we can mitigate the damage by
> > (2). My proposal is as follows:
> >
> > - We provide an anonymous private build-and-test job for all
> contributors.
> > We limit the number of concurrent executors for that job to minimize the
> > financial damage of abuse.
> > - We provide a committer-only private build-and-test job that allows
> > significantly more concurrent executors.
> >
> > Both jobs accept as input a link to a gerrit code review (could be a
> > draft). That way we have a secure audit trail in case of abuse.
> >
> > On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
> >
> > > Before Impala joined the ASF, code reviews had to go through a
> > > "verification" step in which a Jenkins jobs downloaded the patch, ran
> > > all tests and replied back to Jenkins that everything was OK. That
> > > Jenkins job ran on Cloudera infrastructure and could not be accessed,
> > > even in a read-only way, by people outside of Cloudera.
> > >
> > > To follow the Apache way, I am laboring to replace that with a Jenkins
> > > server that can be used by any authorized person and read by any
> > > person. It is at http://jenkins.impala.io:8080. It is able to verify
> > > patches just like the old Jenkins machine. A few remaining questions:
> > >
> > > 1. What should the prerequisites be to turning off the Cloudera-only
> > > Jenkins verification path?
> > >
> > > 2. Who should be able to run jobs on jenkins.impala.io? Some
> > > possibilities: Committers only, anyone who asks, PMC members only,
> > > contributors who ask after submitting 5 patches. Higher bars lead to
> > > less likelihood of abuse, lower ones to easier contributions from
> > > newbies.
> > >
> > > My proposal is this:
> > >
> > > 1. We should turn off the CLoudera-only Jenkins verification path
> > > January 2. jenkins.impala.io is in pretty good shape, and we can make
> > > further improvements as needed. For instance, it took me five minutes
> > > just now to cut down on the not-so-interesting debug output from the
> > > main verification job,
> > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> > >
> > > 2. Everyone with five patches can request a login.
> > >
> > > I'm not married to these ideas, but I wanted to provide a jumping-off
> > > point for discussion.
> > >
> >
>
Re: Jenkins jobs to verify patches
Posted by Taras Bobrovytsky <tb...@cloudera.com>.
I generally agree with Alex's thoughts on this. Just to clarify, the
suggestion is that the anonymous private build-and-test job can be started
without having to log in to Jenkins?
On Tue, Dec 13, 2016 at 4:55 PM, Alex Behm <al...@cloudera.com> wrote:
> The way I see it we have two competing interests that need to be
> reconciled:
> 1. We want to encourage contributions by making it as easy as possible to
> do private test runs. This speaks in favor of allowing any contributor to
> run on jenkins.impala.io.
> 2. Cloudera pays the bill for the EC2 Jenkins workers, so we do need to
> protect Cloudera against abuse.
>
> I am strongly in favor of (1) and I think we can mitigate the damage by
> (2). My proposal is as follows:
>
> - We provide an anonymous private build-and-test job for all contributors.
> We limit the number of concurrent executors for that job to minimize the
> financial damage of abuse.
> - We provide a committer-only private build-and-test job that allows
> significantly more concurrent executors.
>
> Both jobs accept as input a link to a gerrit code review (could be a
> draft). That way we have a secure audit trail in case of abuse.
>
> On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
>
> > Before Impala joined the ASF, code reviews had to go through a
> > "verification" step in which a Jenkins jobs downloaded the patch, ran
> > all tests and replied back to Jenkins that everything was OK. That
> > Jenkins job ran on Cloudera infrastructure and could not be accessed,
> > even in a read-only way, by people outside of Cloudera.
> >
> > To follow the Apache way, I am laboring to replace that with a Jenkins
> > server that can be used by any authorized person and read by any
> > person. It is at http://jenkins.impala.io:8080. It is able to verify
> > patches just like the old Jenkins machine. A few remaining questions:
> >
> > 1. What should the prerequisites be to turning off the Cloudera-only
> > Jenkins verification path?
> >
> > 2. Who should be able to run jobs on jenkins.impala.io? Some
> > possibilities: Committers only, anyone who asks, PMC members only,
> > contributors who ask after submitting 5 patches. Higher bars lead to
> > less likelihood of abuse, lower ones to easier contributions from
> > newbies.
> >
> > My proposal is this:
> >
> > 1. We should turn off the CLoudera-only Jenkins verification path
> > January 2. jenkins.impala.io is in pretty good shape, and we can make
> > further improvements as needed. For instance, it took me five minutes
> > just now to cut down on the not-so-interesting debug output from the
> > main verification job,
> > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> >
> > 2. Everyone with five patches can request a login.
> >
> > I'm not married to these ideas, but I wanted to provide a jumping-off
> > point for discussion.
> >
>
Re: Jenkins jobs to verify patches
Posted by Alex Behm <al...@cloudera.com>.
The way I see it we have two competing interests that need to be reconciled:
1. We want to encourage contributions by making it as easy as possible to
do private test runs. This speaks in favor of allowing any contributor to
run on jenkins.impala.io.
2. Cloudera pays the bill for the EC2 Jenkins workers, so we do need to
protect Cloudera against abuse.
I am strongly in favor of (1) and I think we can mitigate the damage by
(2). My proposal is as follows:
- We provide an anonymous private build-and-test job for all contributors.
We limit the number of concurrent executors for that job to minimize the
financial damage of abuse.
- We provide a committer-only private build-and-test job that allows
significantly more concurrent executors.
Both jobs accept as input a link to a gerrit code review (could be a
draft). That way we have a secure audit trail in case of abuse.
On Tue, Dec 13, 2016 at 1:51 PM, Jim Apple <jb...@cloudera.com> wrote:
> Before Impala joined the ASF, code reviews had to go through a
> "verification" step in which a Jenkins jobs downloaded the patch, ran
> all tests and replied back to Jenkins that everything was OK. That
> Jenkins job ran on Cloudera infrastructure and could not be accessed,
> even in a read-only way, by people outside of Cloudera.
>
> To follow the Apache way, I am laboring to replace that with a Jenkins
> server that can be used by any authorized person and read by any
> person. It is at http://jenkins.impala.io:8080. It is able to verify
> patches just like the old Jenkins machine. A few remaining questions:
>
> 1. What should the prerequisites be to turning off the Cloudera-only
> Jenkins verification path?
>
> 2. Who should be able to run jobs on jenkins.impala.io? Some
> possibilities: Committers only, anyone who asks, PMC members only,
> contributors who ask after submitting 5 patches. Higher bars lead to
> less likelihood of abuse, lower ones to easier contributions from
> newbies.
>
> My proposal is this:
>
> 1. We should turn off the CLoudera-only Jenkins verification path
> January 2. jenkins.impala.io is in pretty good shape, and we can make
> further improvements as needed. For instance, it took me five minutes
> just now to cut down on the not-so-interesting debug output from the
> main verification job,
> http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
>
> 2. Everyone with five patches can request a login.
>
> I'm not married to these ideas, but I wanted to provide a jumping-off
> point for discussion.
>