You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Yibo Cai <yi...@arm.com> on 2021/10/20 04:43:16 UTC
Arm64 github runner
Hi,
We have free Arm64 instances (maintained by Arm) as github action
self-hosted runners for open source projects.
Arrow Arm CI is currently running on Travis. Is an additional Arm64
runner useful? I think we can build and verify Arm64 Linux releases on it.
Yibo
Re: Arm64 github runner
Posted by Yibo Cai <yi...@arm.com>.
Unfortunately the runner may not be available. Will update if things change.
Yibo
On 11/12/21 6:57 PM, Krisztián Szűcs wrote:
> On Wed, Nov 10, 2021 at 2:55 AM Yibo Cai <yi...@arm.com> wrote:
>>
>> Some updates, @kou, @kszucs
>>
>> There are two kinds of runners provided. One is dynamic vm created on
>> demand like travis, suitable for github action runner to verify pr.
>> Another kind is static vm with pre-deployed os, simply an arm64 aws
>> cloud instance. Looks we prefer static vm as a crossbow runner.
>
> Correct. I'm not aware of any way to deploy a github-actions runner to
> ephemeral machines on demand.
>>
>> Please note for security reasons, the runner cannot be accessed directly
>> from internet. Inbound connection requests are rejected. It can initiate
>> connections to other hosts. Is it okay for crossbow?
>
> Shouldn't be a problem, see
> https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
>
>> Any security concern from crossbow side?
>
> Crossbow can only be triggered by trusted contributors, so with
> reasonable precautions we can maintain a low risk setup.
>
>>
>> On 10/22/21 10:07 AM, Yibo Cai wrote:
>>> Thanks. I'm applying for the runner. Will update when ready.
>>>
>>> On 10/22/21 6:31 AM, Krisztián Szűcs wrote:
>>>> On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> It's useful!
>>>>>
>>>>> We have two options to use this:
>>>>>
>>>>> 1. Use this on https://github.com/apache/arrow
>>>>> 2. Use this on https://github.com/ursacomputing/crossbow/
>>>>>
>>>>> 1. is for CI of each commit/pull request.
>>>>>
>>>>> 2. is for CI of daily build and "@github-actions crossbow
>>>>> submit ..." comment in pull request.
>>>>>
>>>>> (We can choose both of them.)
>>>>>
>>>>> If we choose 1., we need to ask INFRA for adding new
>>>>> self-hosted runners. Because we don't have admin permission
>>>>> of apache/arrow. Could you show a URL how to use the
>>>>> self-hosted runners?
>>>>>
>>>>> If we choose 2., we will able to do all needed work by
>>>>> ourselves because we have admin permission of
>>>>> ursacomputing/crossbow/.
>>>> We already have a self-hosted runner configured for crossbow where we
>>>> build the Apple M1 wheels.
>>>>
>>>> I think we should start to configure the new runners for crossbow and
>>>> work out the details, and then later (if we choose to) get the
>>>> required registration tokens from INFRA.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> kou
>>>>>
>>>>> In <2a...@arm.com>
>>>>> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
>>>>> Yibo Cai <yi...@arm.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have free Arm64 instances (maintained by Arm) as github action
>>>>>> self-hosted runners for open source projects.
>>>>>> Arrow Arm CI is currently running on Travis. Is an additional Arm64
>>>>>> runner useful? I think we can build and verify Arm64 Linux releases on
>>>>>> it.
>>>>>>
>>>>>> Yibo
Re: Arm64 github runner
Posted by Martin Grigorov <mg...@apache.org>.
Hi,
On Fri, Nov 12, 2021 at 12:57 PM Krisztián Szűcs <sz...@gmail.com>
wrote:
> On Wed, Nov 10, 2021 at 2:55 AM Yibo Cai <yi...@arm.com> wrote:
> >
> > Some updates, @kou, @kszucs
> >
> > There are two kinds of runners provided. One is dynamic vm created on
> > demand like travis, suitable for github action runner to verify pr.
> > Another kind is static vm with pre-deployed os, simply an arm64 aws
> > cloud instance. Looks we prefer static vm as a crossbow runner.
>
> Correct. I'm not aware of any way to deploy a github-actions runner to
> ephemeral machines on demand.
>
Since you already linked to self hosted runners below I think you are aware
of
https://docs.github.com/en/actions/hosting-your-own-runners/autoscaling-with-self-hosted-runners
This docu page explains that it is possible to do this by using Github
webhooks + "your own autoscaling environment", i.e. you have to implement
it for your needs.
But this setup would require write permissions to the Github repo and this
is something that Infra team won't agree on!
https://docs.github.com/en/actions/hosting-your-own-runners/autoscaling-with-self-hosted-runners#authentication-requirements
"Your access token will require the following scope:
- For private repositories, use an access token with the repo scope
<https://docs.github.com/en/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/#available-scopes>
.
- For public repositories, use an access token with the public_repo scope
<https://docs.github.com/en/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/#available-scopes>
."
public_repo Limits access to public repositories. That includes read/write
access to code, commit statuses, repository projects, collaborators, and
deployment statuses for public repositories and organizations. Also
required for starring public repositories.
>
> > Please note for security reasons, the runner cannot be accessed directly
> > from internet. Inbound connection requests are rejected. It can initiate
> > connections to other hosts. Is it okay for crossbow?
>
> Shouldn't be a problem, see
>
> https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
>
> > Any security concern from crossbow side?
>
> Crossbow can only be triggered by trusted contributors, so with
> reasonable precautions we can maintain a low risk setup.
>
Shameless plug: I've recently blogged how to setup Linux ARM64 self-hosted
runners on Oracle Cloud for free:
https://martin-grigorov.medium.com/github-actions-arm64-runner-on-oracle-cloud-a77cdf7a325a
I hope it is useful to someone!
Regards,
Martin
>
> >
> > On 10/22/21 10:07 AM, Yibo Cai wrote:
> > > Thanks. I'm applying for the runner. Will update when ready.
> > >
> > > On 10/22/21 6:31 AM, Krisztián Szűcs wrote:
> > >> On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com>
> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> It's useful!
> > >>>
> > >>> We have two options to use this:
> > >>>
> > >>> 1. Use this on https://github.com/apache/arrow
> > >>> 2. Use this on https://github.com/ursacomputing/crossbow/
> > >>>
> > >>> 1. is for CI of each commit/pull request.
> > >>>
> > >>> 2. is for CI of daily build and "@github-actions crossbow
> > >>> submit ..." comment in pull request.
> > >>>
> > >>> (We can choose both of them.)
> > >>>
> > >>> If we choose 1., we need to ask INFRA for adding new
> > >>> self-hosted runners. Because we don't have admin permission
> > >>> of apache/arrow. Could you show a URL how to use the
> > >>> self-hosted runners?
> > >>>
> > >>> If we choose 2., we will able to do all needed work by
> > >>> ourselves because we have admin permission of
> > >>> ursacomputing/crossbow/.
> > >> We already have a self-hosted runner configured for crossbow where we
> > >> build the Apple M1 wheels.
> > >>
> > >> I think we should start to configure the new runners for crossbow and
> > >> work out the details, and then later (if we choose to) get the
> > >> required registration tokens from INFRA.
> > >>>
> > >>>
> > >>> Thanks,
> > >>> --
> > >>> kou
> > >>>
> > >>> In <2a...@arm.com>
> > >>> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
> > >>> Yibo Cai <yi...@arm.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> We have free Arm64 instances (maintained by Arm) as github action
> > >>>> self-hosted runners for open source projects.
> > >>>> Arrow Arm CI is currently running on Travis. Is an additional Arm64
> > >>>> runner useful? I think we can build and verify Arm64 Linux releases
> on
> > >>>> it.
> > >>>>
> > >>>> Yibo
>
Re: Arm64 github runner
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Wed, Nov 10, 2021 at 2:55 AM Yibo Cai <yi...@arm.com> wrote:
>
> Some updates, @kou, @kszucs
>
> There are two kinds of runners provided. One is dynamic vm created on
> demand like travis, suitable for github action runner to verify pr.
> Another kind is static vm with pre-deployed os, simply an arm64 aws
> cloud instance. Looks we prefer static vm as a crossbow runner.
Correct. I'm not aware of any way to deploy a github-actions runner to
ephemeral machines on demand.
>
> Please note for security reasons, the runner cannot be accessed directly
> from internet. Inbound connection requests are rejected. It can initiate
> connections to other hosts. Is it okay for crossbow?
Shouldn't be a problem, see
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
> Any security concern from crossbow side?
Crossbow can only be triggered by trusted contributors, so with
reasonable precautions we can maintain a low risk setup.
>
> On 10/22/21 10:07 AM, Yibo Cai wrote:
> > Thanks. I'm applying for the runner. Will update when ready.
> >
> > On 10/22/21 6:31 AM, Krisztián Szűcs wrote:
> >> On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> It's useful!
> >>>
> >>> We have two options to use this:
> >>>
> >>> 1. Use this on https://github.com/apache/arrow
> >>> 2. Use this on https://github.com/ursacomputing/crossbow/
> >>>
> >>> 1. is for CI of each commit/pull request.
> >>>
> >>> 2. is for CI of daily build and "@github-actions crossbow
> >>> submit ..." comment in pull request.
> >>>
> >>> (We can choose both of them.)
> >>>
> >>> If we choose 1., we need to ask INFRA for adding new
> >>> self-hosted runners. Because we don't have admin permission
> >>> of apache/arrow. Could you show a URL how to use the
> >>> self-hosted runners?
> >>>
> >>> If we choose 2., we will able to do all needed work by
> >>> ourselves because we have admin permission of
> >>> ursacomputing/crossbow/.
> >> We already have a self-hosted runner configured for crossbow where we
> >> build the Apple M1 wheels.
> >>
> >> I think we should start to configure the new runners for crossbow and
> >> work out the details, and then later (if we choose to) get the
> >> required registration tokens from INFRA.
> >>>
> >>>
> >>> Thanks,
> >>> --
> >>> kou
> >>>
> >>> In <2a...@arm.com>
> >>> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
> >>> Yibo Cai <yi...@arm.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> We have free Arm64 instances (maintained by Arm) as github action
> >>>> self-hosted runners for open source projects.
> >>>> Arrow Arm CI is currently running on Travis. Is an additional Arm64
> >>>> runner useful? I think we can build and verify Arm64 Linux releases on
> >>>> it.
> >>>>
> >>>> Yibo
Re: Arm64 github runner
Posted by Yibo Cai <yi...@arm.com>.
Some updates, @kou, @kszucs
There are two kinds of runners provided. One is dynamic vm created on
demand like travis, suitable for github action runner to verify pr.
Another kind is static vm with pre-deployed os, simply an arm64 aws
cloud instance. Looks we prefer static vm as a crossbow runner.
Please note for security reasons, the runner cannot be accessed directly
from internet. Inbound connection requests are rejected. It can initiate
connections to other hosts. Is it okay for crossbow? Any security
concern from crossbow side?
On 10/22/21 10:07 AM, Yibo Cai wrote:
> Thanks. I'm applying for the runner. Will update when ready.
>
> On 10/22/21 6:31 AM, Krisztián Szűcs wrote:
>> On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>
>>> Hi,
>>>
>>> It's useful!
>>>
>>> We have two options to use this:
>>>
>>> 1. Use this on https://github.com/apache/arrow
>>> 2. Use this on https://github.com/ursacomputing/crossbow/
>>>
>>> 1. is for CI of each commit/pull request.
>>>
>>> 2. is for CI of daily build and "@github-actions crossbow
>>> submit ..." comment in pull request.
>>>
>>> (We can choose both of them.)
>>>
>>> If we choose 1., we need to ask INFRA for adding new
>>> self-hosted runners. Because we don't have admin permission
>>> of apache/arrow. Could you show a URL how to use the
>>> self-hosted runners?
>>>
>>> If we choose 2., we will able to do all needed work by
>>> ourselves because we have admin permission of
>>> ursacomputing/crossbow/.
>> We already have a self-hosted runner configured for crossbow where we
>> build the Apple M1 wheels.
>>
>> I think we should start to configure the new runners for crossbow and
>> work out the details, and then later (if we choose to) get the
>> required registration tokens from INFRA.
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <2a...@arm.com>
>>> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
>>> Yibo Cai <yi...@arm.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have free Arm64 instances (maintained by Arm) as github action
>>>> self-hosted runners for open source projects.
>>>> Arrow Arm CI is currently running on Travis. Is an additional Arm64
>>>> runner useful? I think we can build and verify Arm64 Linux releases on
>>>> it.
>>>>
>>>> Yibo
Re: Arm64 github runner
Posted by Yibo Cai <yi...@arm.com>.
Thanks. I'm applying for the runner. Will update when ready.
On 10/22/21 6:31 AM, Krisztián Szűcs wrote:
> On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>>
>> Hi,
>>
>> It's useful!
>>
>> We have two options to use this:
>>
>> 1. Use this on https://github.com/apache/arrow
>> 2. Use this on https://github.com/ursacomputing/crossbow/
>>
>> 1. is for CI of each commit/pull request.
>>
>> 2. is for CI of daily build and "@github-actions crossbow
>> submit ..." comment in pull request.
>>
>> (We can choose both of them.)
>>
>> If we choose 1., we need to ask INFRA for adding new
>> self-hosted runners. Because we don't have admin permission
>> of apache/arrow. Could you show a URL how to use the
>> self-hosted runners?
>>
>> If we choose 2., we will able to do all needed work by
>> ourselves because we have admin permission of
>> ursacomputing/crossbow/.
> We already have a self-hosted runner configured for crossbow where we
> build the Apple M1 wheels.
>
> I think we should start to configure the new runners for crossbow and
> work out the details, and then later (if we choose to) get the
> required registration tokens from INFRA.
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In <2a...@arm.com>
>> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
>> Yibo Cai <yi...@arm.com> wrote:
>>
>>> Hi,
>>>
>>> We have free Arm64 instances (maintained by Arm) as github action
>>> self-hosted runners for open source projects.
>>> Arrow Arm CI is currently running on Travis. Is an additional Arm64
>>> runner useful? I think we can build and verify Arm64 Linux releases on
>>> it.
>>>
>>> Yibo
Re: Arm64 github runner
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Oct 21, 2021 at 11:53 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> Hi,
>
> It's useful!
>
> We have two options to use this:
>
> 1. Use this on https://github.com/apache/arrow
> 2. Use this on https://github.com/ursacomputing/crossbow/
>
> 1. is for CI of each commit/pull request.
>
> 2. is for CI of daily build and "@github-actions crossbow
> submit ..." comment in pull request.
>
> (We can choose both of them.)
>
> If we choose 1., we need to ask INFRA for adding new
> self-hosted runners. Because we don't have admin permission
> of apache/arrow. Could you show a URL how to use the
> self-hosted runners?
>
> If we choose 2., we will able to do all needed work by
> ourselves because we have admin permission of
> ursacomputing/crossbow/.
We already have a self-hosted runner configured for crossbow where we
build the Apple M1 wheels.
I think we should start to configure the new runners for crossbow and
work out the details, and then later (if we choose to) get the
required registration tokens from INFRA.
>
>
> Thanks,
> --
> kou
>
> In <2a...@arm.com>
> "Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
> Yibo Cai <yi...@arm.com> wrote:
>
> > Hi,
> >
> > We have free Arm64 instances (maintained by Arm) as github action
> > self-hosted runners for open source projects.
> > Arrow Arm CI is currently running on Travis. Is an additional Arm64
> > runner useful? I think we can build and verify Arm64 Linux releases on
> > it.
> >
> > Yibo
Re: Arm64 github runner
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
It's useful!
We have two options to use this:
1. Use this on https://github.com/apache/arrow
2. Use this on https://github.com/ursacomputing/crossbow/
1. is for CI of each commit/pull request.
2. is for CI of daily build and "@github-actions crossbow
submit ..." comment in pull request.
(We can choose both of them.)
If we choose 1., we need to ask INFRA for adding new
self-hosted runners. Because we don't have admin permission
of apache/arrow. Could you show a URL how to use the
self-hosted runners?
If we choose 2., we will able to do all needed work by
ourselves because we have admin permission of
ursacomputing/crossbow/.
Thanks,
--
kou
In <2a...@arm.com>
"Arm64 github runner" on Wed, 20 Oct 2021 12:43:16 +0800,
Yibo Cai <yi...@arm.com> wrote:
> Hi,
>
> We have free Arm64 instances (maintained by Arm) as github action
> self-hosted runners for open source projects.
> Arrow Arm CI is currently running on Travis. Is an additional Arm64
> runner useful? I think we can build and verify Arm64 Linux releases on
> it.
>
> Yibo