You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Mick Semb Wever <mc...@apache.org> on 2021/04/06 10:34:16 UTC

Better stability with docker authenticated jenkins agents

tl;dr
Can and should all jenkins agents be (automatically) docker authenticated,
for improved stability around docker commands?


This past week the ci-cassandra.apache.org CI fell over because a fair
percentage of docker pulls failed. Our pipeline runs a lot of docker
containers. In the past week the number of containers run went from ~180 to
~270 and that pushed something over the edge. All of a sudden all docker
commands had a significant chance of failing, that was high enough to
ensure every pipeline was guaranteed to fail. These failures came in a few
different forms, the list can be read in INFRA-21666. Googling them shows
that this is a known problem, sometimes around firewalls, networks, dns,
etc. Our jenkins agents are donated by a handful of different companies and
are located in various different places, so such issues don't make much
sense. The other typical fix reported was to just run docker authenticated,
i.e. `docker login`. Trying this immediately solved all problems on
ci-cassandra.apache.org. This was done with a temporary (and empty)
dockerhub account, that each agent has manually logged in with. Based on
all this, the request has been made for an official apache CI
dockerhub account and to have jenkins agents automatically logged in, with
credentials stored in an appropriate manner.

Has anyone experience with such issues before?
Is this a sound and reasonable request to ask of Infra?

regards,
Mick

Re: Better stability with docker authenticated jenkins agents

Posted by Matt Sicker <bo...@gmail.com>.
Folders can store credentials as well if you're on the main Jenkins
instance here.

On Wed, 14 Apr 2021 at 12:19, Joan Touzet <wo...@apache.org> wrote:
>
> On 14/04/2021 13:05, Mick Semb Wever wrote:
> >> Then it is easy to use our own Jenkins-stored creds
> >> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jenkins.io_doc_book_using_using-2Dcredentials_&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=42Z7FyMoAS1DbvgKNjU8zxi7xTPVAGalPzk7bfmRVgw&m=Oa7qm3rmqlXgK8wUUTEnADpIgZT18tE6W-ghG1QUspU&s=vRKVrs-iZa60eTjfvdc1aggzj_-CyObd2xs70PdN1rs&e= ), or Infra can
> >> give us a pair to use instead.
> >>
> >
> >
> > Ahh! Do we have rights to store credentials in the ASF jenkins? I was
> > under the impression INFRA had to do this for us.
>
> Yes, you can, especially for Cassandra where you have your own separate
> build master.
>
> Projects are on the honour system not to use creds they're not supposed
> to have access to, which is at least one major reason why CouchDB
> requested and received their own CloudBees/Jenkins build master. The
> separate build master structure should prevent anyone outside of your
> PMC-approved group from creating any jobs on your build server that
> might use those credentials inappropriately.
>
> -Joan

Re: Better stability with docker authenticated jenkins agents

Posted by Joan Touzet <wo...@apache.org>.
On 14/04/2021 13:05, Mick Semb Wever wrote:
>> Then it is easy to use our own Jenkins-stored creds
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jenkins.io_doc_book_using_using-2Dcredentials_&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=42Z7FyMoAS1DbvgKNjU8zxi7xTPVAGalPzk7bfmRVgw&m=Oa7qm3rmqlXgK8wUUTEnADpIgZT18tE6W-ghG1QUspU&s=vRKVrs-iZa60eTjfvdc1aggzj_-CyObd2xs70PdN1rs&e= ), or Infra can
>> give us a pair to use instead.
>>
> 
> 
> Ahh! Do we have rights to store credentials in the ASF jenkins? I was
> under the impression INFRA had to do this for us.

Yes, you can, especially for Cassandra where you have your own separate
build master.

Projects are on the honour system not to use creds they're not supposed
to have access to, which is at least one major reason why CouchDB
requested and received their own CloudBees/Jenkins build master. The
separate build master structure should prevent anyone outside of your
PMC-approved group from creating any jobs on your build server that
might use those credentials inappropriately.

-Joan

Re: Better stability with docker authenticated jenkins agents

Posted by Mick Semb Wever <mc...@apache.org>.
> Then it is easy to use our own Jenkins-stored creds
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jenkins.io_doc_book_using_using-2Dcredentials_&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=42Z7FyMoAS1DbvgKNjU8zxi7xTPVAGalPzk7bfmRVgw&m=Oa7qm3rmqlXgK8wUUTEnADpIgZT18tE6W-ghG1QUspU&s=vRKVrs-iZa60eTjfvdc1aggzj_-CyObd2xs70PdN1rs&e= ), or Infra can
> give us a pair to use instead.
>


Ahh! Do we have rights to store credentials in the ASF jenkins? I was
under the impression INFRA had to do this for us.
This goes to the crux of the problem raised: the original request on
INFRA, i.e. to have "credentials stored in an appropriate manner".

Gavin, is it an acceptable approach to have projects create their own
dockerhub accounts and to store those credentials in jenkins
themselves as a way to solve the rate-limiting of apache/* namespace
images?

Re: Better stability with docker authenticated jenkins agents

Posted by Joan Touzet <wo...@apache.org>.
On 07/04/2021 07:43, Mick Semb Wever wrote:
>>
>> My understanding is that pulls of all images from the apache/* namespace
>> are not subject to rate limiting. Thus, the recommendation to move
>> everything you need inside of it.
>>
> As >95% of our CI docker commands are pulls from apache/ images,
> if rate-limiting is the cause of this (note that nowhere did we see the
> toomanyrequests response error),
> then we still need to authenticate docker to get the rate-limiting
> exception for those apache/ images,
> as you mention here¹. Has that changed?

My mistake, I mis-remembered why we wanted this change. Moving images to
the apache organisation only prevents them from being auto-deleted by
Docker Hub's scrubbing process, which started in the middle of last year.

Your proposal to have a Docker Hub account seems reasonable, but since
this is something projects can solve themselves, not critical path.

We'll probably store the Docker Hub creds in Jenkins, then reference
that in the build, which is what we do for other creds we need today. As
we use declarative pipeline, that's something like:

              docker {
                image "${DOCKER_IMAGE}"
                label 'docker'
                args "${DOCKER_ARGS}"
                registryCredentialsId "${DOCKER_CREDS"}
              }

Then it is easy to use our own Jenkins-stored creds
(https://www.jenkins.io/doc/book/using/using-credentials/), or Infra can
give us a pair to use instead.

> Maybe I'm circling, but doesn't this then support the need that we should
> have jenkins agents docker authenticated somehow?
> 
> [1]
> https://lists.apache.org/thread.html/rede9074dd499ae10dcb501dedcdec43fe9cbb5c646a2c38b19946f85%40%3Cbuilds.apache.org%3E

Re: Better stability with docker authenticated jenkins agents

Posted by Mick Semb Wever <mc...@apache.org>.
>
> My understanding is that pulls of all images from the apache/* namespace
> are not subject to rate limiting. Thus, the recommendation to move
> everything you need inside of it.
>


As >95% of our CI docker commands are pulls from apache/ images,
if rate-limiting is the cause of this (note that nowhere did we see the
toomanyrequests response error),
then we still need to authenticate docker to get the rate-limiting
exception for those apache/ images,
as you mention here¹. Has that changed?

Maybe I'm circling, but doesn't this then support the need that we should
have jenkins agents docker authenticated somehow?


[1]
https://lists.apache.org/thread.html/rede9074dd499ae10dcb501dedcdec43fe9cbb5c646a2c38b19946f85%40%3Cbuilds.apache.org%3E

Re: Better stability with docker authenticated jenkins agents

Posted by Joan Touzet <wo...@apache.org>.
On 06/04/2021 18:09, Mick Semb Wever wrote:
>>
>>> Has anyone experience with such issues before?
>>
>> Yes. See https://issues.apache.org/jira/browse/INFRA-20795 for detail.
>>
>> In short, once Infra agrees to create the images for us, we'll move all
>> our CI dependencies into those containers, and should no longer have
>> issues.
>>
> 
> 
> Thanks Joan. We do have our testing images already in the apache docker
> account. And the unauthenticated docker commands were falling over even
> pulling just official ubuntu images.

My understanding is that pulls of all images from the apache/* namespace
are not subject to rate limiting. Thus, the recommendation to move
everything you need inside of it.

If that's not practical, or you're building images that require other
assets... this won't work for you.

-Joan

Re: Better stability with docker authenticated jenkins agents

Posted by Mick Semb Wever <mc...@apache.org>.
>
> > Has anyone experience with such issues before?
>
> Yes. See https://issues.apache.org/jira/browse/INFRA-20795 for detail.
>
> In short, once Infra agrees to create the images for us, we'll move all
> our CI dependencies into those containers, and should no longer have
> issues.
>


Thanks Joan. We do have our testing images already in the apache docker
account. And the unauthenticated docker commands were falling over even
pulling just official ubuntu images.



> > Is this a sound and reasonable request to ask of Infra?
>
> This could work too but *only* when Infra manages the agents.
>


I was wondering if secrets could be stored into Jenkins, but yeah unmanaged
agents 😒

Docker accounts also come with api tokens, so different tokens might be
handed out to the different jenkins clusters. But I really don't know
what's best here, especially with little protection over `.docker/config
.json` … 🤷🏻‍♀️

Re: Better stability with docker authenticated jenkins agents

Posted by Joan Touzet <wo...@apache.org>.
Hi Mick,

On 06/04/2021 06:34, Mick Semb Wever wrote:
> tl;dr
> Can and should all jenkins agents be (automatically) docker authenticated,
> for improved stability around docker commands?
> 
> 
> This past week the ci-cassandra.apache.org CI fell over because a fair
> percentage of docker pulls failed. Our pipeline runs a lot of docker
> containers. In the past week the number of containers run went from ~180 to
> ~270 and that pushed something over the edge. All of a sudden all docker
> commands had a significant chance of failing, that was high enough to
> ensure every pipeline was guaranteed to fail. These failures came in a few
> different forms, the list can be read in INFRA-21666. Googling them shows
> that this is a known problem, sometimes around firewalls, networks, dns,
> etc. Our jenkins agents are donated by a handful of different companies and
> are located in various different places, so such issues don't make much
> sense. The other typical fix reported was to just run docker authenticated,
> i.e. `docker login`. Trying this immediately solved all problems on
> ci-cassandra.apache.org. This was done with a temporary (and empty)
> dockerhub account, that each agent has manually logged in with. Based on
> all this, the request has been made for an official apache CI
> dockerhub account and to have jenkins agents automatically logged in, with
> credentials stored in an appropriate manner.
> 
> Has anyone experience with such issues before?

Yes. See https://issues.apache.org/jira/browse/INFRA-20795 for detail.

In short, once Infra agrees to create the images for us, we'll move all
our CI dependencies into those containers, and should no longer have issues.

> Is this a sound and reasonable request to ask of Infra?

This could work too but *only* when Infra manages the agents. For most
of our agents, Infra does not have ssh access (due to NAT-style
proxying) so there's nothing they can do... _other_ than give us those
dockerhub creds. That sounds more unwieldy than the approach outlined in
ticket 20795.

-Joan