You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2022/01/21 17:06:54 UTC
[CI] Fwd: dockerhub limitations - maximum user count reached
For the record, here is a message from INFRA. I don't remember if we
use an Apache account at Docker Hub?
-------- Message transféré --------
Sujet : dockerhub limitations - maximum user count reached
Date : Fri, 21 Jan 2022 08:56:45 -0800
De : Chris Lambertus <cm...@apache.org>
Répondre à : builds@apache.org
Pour : builds <bu...@apache.org>
Hi folks,
We have reached our 200 seat maximum for dockerhub users in the Apache
organization, and Docker Hub is unable to provide any activity records
for those users, so we do not know who to cull.
Do we have any internal contacts amongst the list at DockerHub who might
be able to assist?
If not, we'll need to start removing users at more-or-less random as needed.
Infra will be moving to a more conservative process for adding dockerhub
users in the future. We will request that:
- autobuilds be used where possible instead of manual creation
- user access to dockerhub/teams be limited to a max of 2 per project
- all use of dockerhub for pushing artifacts from build servers be
limited to the Infra-provided accounts for this purpose (e.g don't use
personal accounts to push artifacts)
If your project falls into any of the above categories, please let us
know ASAP so we can trim excess users.
Thanks,
Chris
ASF Infra
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
In <53...@python.org>
"Re: [CI] Fwd: dockerhub limitations - maximum user count reached" on Sat, 22 Jan 2022 12:27:20 +0100,
Antoine Pitrou <an...@python.org> wrote:
>> It may be better that we use ghcr.io (Docker registry
>> provided by GitHub) instead of Docker Hub. Because we use
>> our Docker images only for CI. They are not intended to be
>> used by users.
>
> Perhaps by "users" you mean non-Arrow developers?
Yes.
> The Docker images are definitely useful to reproduce
> issues locally, or to run certain builds.
Developers can still use Docker images that are used for CI
with ghcr.io. And developer's workflow isn't changed.
Developers use Archery or raw docker-compose.yml. Docker
image URLs (including ghcr.io/) are embedded in
docker-compose.yml. So developers don't need to care about
Docker image URLs. Developers don't need to change their
command lines.
Thanks,
--
kou
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Jarek Potiuk <ja...@potiuk.com>.
>
>
> It may be better that we use ghcr.io (Docker registry
> provided by GitHub) instead of Docker Hub. Because we use
> our Docker images only for CI. They are not intended to be
> used by users. All Apache Arrow committers will be able to
> push ghcr.io/apache/arrow.
>
>
Just a comment from a friendly Airflow Maintainer - absolutely. Use Ghcr.io
for that.
We were very unhappy with DockerHub used in CI due to slowness, delays and
the need to have separate authentication tokens. We switched to ghcr.io in
Airflow immediately when they were available (even when they were in beta)
and it works very well since. It can be very nicely integrated with GitHub
Actions (temporary tokens in "main" / "workflow_run" / "workflow_targe"
builds might be used to securely get "selective" write access only to the
registry. The registry has a nice API (for example allowing retention of
old images).
J.
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Phillip Cloud <cp...@gmail.com>.
+1 for using ghcr.io. It integrates well with GitHub (e.g., the packages
feature) and is generally a better user experience than dockerhub.
On Sat, Jan 22, 2022 at 6:27 AM Antoine Pitrou <an...@python.org> wrote:
>
> Le 22/01/2022 à 05:20, Sutou Kouhei a écrit :
> > Hi,
> >
> > We use asfjenkins account on apache/arrow but we use my
> > account or Krisztián's account on ursacomputing/crossbow for
> > Crossbow. (We set Docker Hub account information on GitHub
> > Actions and Azure Pipelines for ursacomputing/crossbow.)
> >
> > It may be better that we use ghcr.io (Docker registry
> > provided by GitHub) instead of Docker Hub. Because we use
> > our Docker images only for CI. They are not intended to be
> > used by users.
>
> Perhaps by "users" you mean non-Arrow developers? The Docker images are
> definitely useful to reproduce issues locally, or to run certain builds.
>
You can enable selective permissions, like public reads and private writes,
so anyone who wants
to can pull the images, while not just anyone can push to the registry.
>
> Regards
>
> Antoine.
>
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Antoine Pitrou <an...@python.org>.
Le 22/01/2022 à 05:20, Sutou Kouhei a écrit :
> Hi,
>
> We use asfjenkins account on apache/arrow but we use my
> account or Krisztián's account on ursacomputing/crossbow for
> Crossbow. (We set Docker Hub account information on GitHub
> Actions and Azure Pipelines for ursacomputing/crossbow.)
>
> It may be better that we use ghcr.io (Docker registry
> provided by GitHub) instead of Docker Hub. Because we use
> our Docker images only for CI. They are not intended to be
> used by users.
Perhaps by "users" you mean non-Arrow developers? The Docker images are
definitely useful to reproduce issues locally, or to run certain builds.
Regards
Antoine.
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
We use asfjenkins account on apache/arrow but we use my
account or Krisztián's account on ursacomputing/crossbow for
Crossbow. (We set Docker Hub account information on GitHub
Actions and Azure Pipelines for ursacomputing/crossbow.)
It may be better that we use ghcr.io (Docker registry
provided by GitHub) instead of Docker Hub. Because we use
our Docker images only for CI. They are not intended to be
used by users. All Apache Arrow committers will be able to
push ghcr.io/apache/arrow.
BTW, we only need my account and Krisztián's account for
now. It seems that we have 3 accounts:
* kszucs (Krisztián's account)
* pitrou (Antoine's account)
* ktou (my account)
https://issues.apache.org/jira/browse/INFRA-19499?focusedCommentId=16989793&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16989793
We can use only one Docker Hub account for our use case by
updating secrets on GitHub Actions and Azure Pipelines for
ursacomputing/crossbow. So we can remove at most two
Docker Hub accounts from the Apache organization .
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Re: [CI] Fwd: dockerhub limitations - maximum user count reached" on Fri, 21 Jan 2022 21:54:29 +0100,
Krisztián Szűcs <sz...@gmail.com> wrote:
> On Fri, Jan 21, 2022 at 6:07 PM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> For the record, here is a message from INFRA. I don't remember if we
>> use an Apache account at Docker Hub?
>
> Actually we're heavy users of the apache/arrow-dev docker repository
> [1] counting 15084 images (which we should probably clean up).
> If I recall correctly INFRA has set up the credentials for us with
> asfjenkins user. Additionally a couple of us are members, but I can't
> see a list of users on the interface.
>
>> Docker Hub is unable to provide any activity records for those users, so we do not know who to cull.
>
> There is an activity tab on dockerhub, but the browser is unable to
> load the data so I cannot provide more information.
>
> [1]: https://hub.docker.com/r/apache/arrow-dev/tags
>
>>
>>
>> -------- Message transféré --------
>> Sujet : dockerhub limitations - maximum user count reached
>> Date : Fri, 21 Jan 2022 08:56:45 -0800
>> De : Chris Lambertus <cm...@apache.org>
>> Répondre à : builds@apache.org
>> Pour : builds <bu...@apache.org>
>>
>> Hi folks,
>>
>> We have reached our 200 seat maximum for dockerhub users in the Apache
>> organization, and Docker Hub is unable to provide any activity records
>> for those users, so we do not know who to cull.
>>
>> Do we have any internal contacts amongst the list at DockerHub who might
>> be able to assist?
>>
>> If not, we'll need to start removing users at more-or-less random as needed.
>>
>> Infra will be moving to a more conservative process for adding dockerhub
>> users in the future. We will request that:
>>
>> - autobuilds be used where possible instead of manual creation
>> - user access to dockerhub/teams be limited to a max of 2 per project
>> - all use of dockerhub for pushing artifacts from build servers be
>> limited to the Infra-provided accounts for this purpose (e.g don't use
>> personal accounts to push artifacts)
>>
>> If your project falls into any of the above categories, please let us
>> know ASAP so we can trim excess users.
>>
>> Thanks,
>> Chris
>> ASF Infra
>>
>>
>>
Re: [CI] Fwd: dockerhub limitations - maximum user count reached
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Fri, Jan 21, 2022 at 6:07 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> For the record, here is a message from INFRA. I don't remember if we
> use an Apache account at Docker Hub?
Actually we're heavy users of the apache/arrow-dev docker repository
[1] counting 15084 images (which we should probably clean up).
If I recall correctly INFRA has set up the credentials for us with
asfjenkins user. Additionally a couple of us are members, but I can't
see a list of users on the interface.
> Docker Hub is unable to provide any activity records for those users, so we do not know who to cull.
There is an activity tab on dockerhub, but the browser is unable to
load the data so I cannot provide more information.
[1]: https://hub.docker.com/r/apache/arrow-dev/tags
>
>
> -------- Message transféré --------
> Sujet : dockerhub limitations - maximum user count reached
> Date : Fri, 21 Jan 2022 08:56:45 -0800
> De : Chris Lambertus <cm...@apache.org>
> Répondre à : builds@apache.org
> Pour : builds <bu...@apache.org>
>
> Hi folks,
>
> We have reached our 200 seat maximum for dockerhub users in the Apache
> organization, and Docker Hub is unable to provide any activity records
> for those users, so we do not know who to cull.
>
> Do we have any internal contacts amongst the list at DockerHub who might
> be able to assist?
>
> If not, we'll need to start removing users at more-or-less random as needed.
>
> Infra will be moving to a more conservative process for adding dockerhub
> users in the future. We will request that:
>
> - autobuilds be used where possible instead of manual creation
> - user access to dockerhub/teams be limited to a max of 2 per project
> - all use of dockerhub for pushing artifacts from build servers be
> limited to the Infra-provided accounts for this purpose (e.g don't use
> personal accounts to push artifacts)
>
> If your project falls into any of the above categories, please let us
> know ASAP so we can trim excess users.
>
> Thanks,
> Chris
> ASF Infra
>
>
>