You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ganesh Raju <ga...@linaro.org> on 2020/02/19 21:37:59 UTC
CI setup on dedicated arm hardware
Hi,
I am following up on the discussion from here
<https://github.com/apache/arrow/pull/6253>, with interest to have
dedicated arm hardware for CI setup. We can surely help with that if we get
a go-ahead from the project.
Thanks,
Ganesh
--
IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Dedicated hardware for Arrow CI / benchmarking
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,
Workflow triggered from issue_comment may not have enough
permission (write permission) in pull request from other
repository.
I considered how to implement comment bot by GitHub Actions
before: https://issues.apache.org/jira/browse/ARROW-7203
It may help you.
Thanks,
--
kou
In <CA...@mail.gmail.com>
"Re: Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup on dedicated arm hardware]" on Thu, 5 Mar 2020 17:29:03 +0100,
Krisztián Szűcs <sz...@gmail.com> wrote:
> On Thu, Mar 5, 2020 at 4:27 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> On Thu, Mar 5, 2020 at 5:01 AM Krisztián Szűcs
>> <sz...@gmail.com> wrote:
>> >
>> > On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <an...@python.org> wrote:
>> > >
>> > >
>> > > It sounds like this would be a good reason to use BuildKite, which AFAIU
>> > > can automatically provision and operate cloud resources for us?
>> > That's true for buildbot as well [1] including openstack and mesos support which
>> > would be nice for physical machines.
>>
>> I think we should try to reduce the number of services we have to
>> configure and sysadmin, it has a non-zero cost in time and money.
> Agree.
>>
>> > With the ursa machines down we ended up with two missing services:
>> > - nightly trigger and report: I've already ported the cron jobs triggering the
>> > builds and reporting to github actions [2]
>> > - the comment bot: it is implemented in ursabot including the github event
>> > listener, so buildkite wouldn't provide a solution here because we need
>> > to maintain a web service for it.
>> >
>> > The solution for the comment bot could be to factor out the implementation
>> > from ursabot and run it from a github actions build triggered using the
>> > issue_comment event [3] (buildkite doesn't seem to support it currently [4]).
> Luckily it is going to work. Since github has added support for additional
> action trigger we can act on issue and pull request review comments.
> I've tested it on my fork [1]. This means that we can implement the comment
> bot without maintaining any service. I'm working on it.
>
> [1]: https://github.com/kszucs/arrow/runs/488092876?check_suite_focus=true#step:5:8
>> >
>> > We can also host the buildbot buildmaster in the cloud, at least until we
>> > decomission all of its services.
>> >
>> > [1] https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers
>> > [2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows
>> > [3] https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment
>> > [4] https://github.com/buildkite/feedback/issues/288
>> > >
>> > >
>> > > Le 04/03/2020 à 16:21, Wes McKinney a écrit :
>> > > > hi folks,
>> > > >
>> > > > The tornado the night before last in Nashville, Tennessee temporarily
>> > > > disabled the physical hardware that I have been running there where
>> > > > we've been running "Ursabot" builds and where we've been experimenting
>> > > > with other self-hosted CI solutions like GitHub Actions Self-Hosted
>> > > > Runners and Buildkite.
>> > > >
>> > > > While dedicated physical hardware can be useful to reduce cloud
>> > > > computing costs, I think this natural disaster should help inform our
>> > > > approach to this problem:
>> > > >
>> > > > * In the event that physical hosted infrastructure becomes
>> > > > unavailable, we eventually should have the capability to spin up
>> > > > machines in the cloud with the desired properties (GCE provides both
>> > > > Linux and Windows VMs, for example)
>> > > > * Adding new machines to our CI process ideally should not require a
>> > > > human in-the-loop (GHA presently requires a human -- in particular
>> > > > someone from ASF Infra -- in the loop to add workers, so this IMHO
>> > > > should be taken into consideration)
>> > > >
>> > > > Any other thoughts about this topic would be welcome.
>> > > >
>> > > > Thanks
>> > > > Wes
>> > > >
>> > > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
>> > > > <sz...@gmail.com> wrote:
>> > > >>
>> > > >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
>> > > >>>
>> > > >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
>> > > >>> <sz...@gmail.com> wrote:
>> > > >>>>
>> > > >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
>> > > >>>>>
>> > > >>>>> hi Ganesh,
>> > > >>>>>
>> > > >>>>> Thanks for writing.
>> > > >>>>>
>> > > >>>>> I've been working on setting up Buildkite (BK) as a way for third
>> > > >>>>> parties for attach machines to run builds on, with a free organization
>> > > >>>>> at
>> > > >>>>>
>> > > >>>>> https://buildkite.com/apache-arrow
>> > > >>>>>
>> > > >>>>> Configuring a new machine to accept builds is very easy [1] and takes
>> > > >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work
>> > > >>>>> on Windows). Currently I've attached 6 machines:
>> > > >>>>>
>> > > >>>>> * 2 CUDA-capable Linux x86
>> > > >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable
>> > > >>>>> * 1 macOS
>> > > >>>>>
>> > > >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds
>> > > >>>>> triggered in BK can report commit statuses on GitHub [2]
>> > > >>>>>
>> > > >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this
>> > > >>>>> also but the workflow for new machines to be contributed needs to be
>> > > >>>>> proven out.
>> > > >>>> I've already tried it out, and setting up self-hosted github runners is just as
>> > > >>>> easy as with buildkite, drawbacks:
>> > > >>>
>> > > >>> I don't mean to be argumentative, but I don't see how this can be true
>> > > >>> if we don't have access to the "Settings" tab on GitHub
>> > > >> On a fork where I have access for that tab.
>> > > >>>
>> > > >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
>> > > >>>
>> > > >>>> - I'm unsure how would the tagging selection work in practice [1]
>> > > >>>> - We won't have access to the runners dashboard in lack of admin rights
>> > > >>>> for the apache/arrow repository - so we need to test out the workflow.
>> > > >>>>
>> > > >>>> I've created an INFRA ticket to get some information and to track it:
>> > > >>>> https://issues.apache.org/jira/browse/INFRA-19875
>> > > >>>>
>> > > >>>> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
>> > > >>>>>
>> > > >>>>> Thanks,
>> > > >>>>> Wes
>> > > >>>>>
>> > > >>>>> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
>> > > >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217
>> > > >>>>>
>> > > >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
>> > > >>>>>>
>> > > >>>>>> Hi,
>> > > >>>>>> I am following up on the discussion from here
>> > > >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have
>> > > >>>>>> dedicated arm hardware for CI setup. We can surely help with that if we get
>> > > >>>>>> a go-ahead from the project.
>> > > >>>>>>
>> > > >>>>>> Thanks,
>> > > >>>>>> Ganesh
>> > > >>>>>>
>> > > >>>>>> --
>> > > >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup
on dedicated arm hardware]
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Mar 5, 2020 at 4:27 PM Wes McKinney <we...@gmail.com> wrote:
>
> On Thu, Mar 5, 2020 at 5:01 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <an...@python.org> wrote:
> > >
> > >
> > > It sounds like this would be a good reason to use BuildKite, which AFAIU
> > > can automatically provision and operate cloud resources for us?
> > That's true for buildbot as well [1] including openstack and mesos support which
> > would be nice for physical machines.
>
> I think we should try to reduce the number of services we have to
> configure and sysadmin, it has a non-zero cost in time and money.
Agree.
>
> > With the ursa machines down we ended up with two missing services:
> > - nightly trigger and report: I've already ported the cron jobs triggering the
> > builds and reporting to github actions [2]
> > - the comment bot: it is implemented in ursabot including the github event
> > listener, so buildkite wouldn't provide a solution here because we need
> > to maintain a web service for it.
> >
> > The solution for the comment bot could be to factor out the implementation
> > from ursabot and run it from a github actions build triggered using the
> > issue_comment event [3] (buildkite doesn't seem to support it currently [4]).
Luckily it is going to work. Since github has added support for additional
action trigger we can act on issue and pull request review comments.
I've tested it on my fork [1]. This means that we can implement the comment
bot without maintaining any service. I'm working on it.
[1]: https://github.com/kszucs/arrow/runs/488092876?check_suite_focus=true#step:5:8
> >
> > We can also host the buildbot buildmaster in the cloud, at least until we
> > decomission all of its services.
> >
> > [1] https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers
> > [2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows
> > [3] https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment
> > [4] https://github.com/buildkite/feedback/issues/288
> > >
> > >
> > > Le 04/03/2020 à 16:21, Wes McKinney a écrit :
> > > > hi folks,
> > > >
> > > > The tornado the night before last in Nashville, Tennessee temporarily
> > > > disabled the physical hardware that I have been running there where
> > > > we've been running "Ursabot" builds and where we've been experimenting
> > > > with other self-hosted CI solutions like GitHub Actions Self-Hosted
> > > > Runners and Buildkite.
> > > >
> > > > While dedicated physical hardware can be useful to reduce cloud
> > > > computing costs, I think this natural disaster should help inform our
> > > > approach to this problem:
> > > >
> > > > * In the event that physical hosted infrastructure becomes
> > > > unavailable, we eventually should have the capability to spin up
> > > > machines in the cloud with the desired properties (GCE provides both
> > > > Linux and Windows VMs, for example)
> > > > * Adding new machines to our CI process ideally should not require a
> > > > human in-the-loop (GHA presently requires a human -- in particular
> > > > someone from ASF Infra -- in the loop to add workers, so this IMHO
> > > > should be taken into consideration)
> > > >
> > > > Any other thoughts about this topic would be welcome.
> > > >
> > > > Thanks
> > > > Wes
> > > >
> > > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
> > > > <sz...@gmail.com> wrote:
> > > >>
> > > >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
> > > >>>
> > > >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
> > > >>> <sz...@gmail.com> wrote:
> > > >>>>
> > > >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> > > >>>>>
> > > >>>>> hi Ganesh,
> > > >>>>>
> > > >>>>> Thanks for writing.
> > > >>>>>
> > > >>>>> I've been working on setting up Buildkite (BK) as a way for third
> > > >>>>> parties for attach machines to run builds on, with a free organization
> > > >>>>> at
> > > >>>>>
> > > >>>>> https://buildkite.com/apache-arrow
> > > >>>>>
> > > >>>>> Configuring a new machine to accept builds is very easy [1] and takes
> > > >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work
> > > >>>>> on Windows). Currently I've attached 6 machines:
> > > >>>>>
> > > >>>>> * 2 CUDA-capable Linux x86
> > > >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable
> > > >>>>> * 1 macOS
> > > >>>>>
> > > >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds
> > > >>>>> triggered in BK can report commit statuses on GitHub [2]
> > > >>>>>
> > > >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this
> > > >>>>> also but the workflow for new machines to be contributed needs to be
> > > >>>>> proven out.
> > > >>>> I've already tried it out, and setting up self-hosted github runners is just as
> > > >>>> easy as with buildkite, drawbacks:
> > > >>>
> > > >>> I don't mean to be argumentative, but I don't see how this can be true
> > > >>> if we don't have access to the "Settings" tab on GitHub
> > > >> On a fork where I have access for that tab.
> > > >>>
> > > >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
> > > >>>
> > > >>>> - I'm unsure how would the tagging selection work in practice [1]
> > > >>>> - We won't have access to the runners dashboard in lack of admin rights
> > > >>>> for the apache/arrow repository - so we need to test out the workflow.
> > > >>>>
> > > >>>> I've created an INFRA ticket to get some information and to track it:
> > > >>>> https://issues.apache.org/jira/browse/INFRA-19875
> > > >>>>
> > > >>>> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Wes
> > > >>>>>
> > > >>>>> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> > > >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217
> > > >>>>>
> > > >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>> I am following up on the discussion from here
> > > >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have
> > > >>>>>> dedicated arm hardware for CI setup. We can surely help with that if we get
> > > >>>>>> a go-ahead from the project.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Ganesh
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup
on dedicated arm hardware]
Posted by Wes McKinney <we...@gmail.com>.
On Thu, Mar 5, 2020 at 5:01 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <an...@python.org> wrote:
> >
> >
> > It sounds like this would be a good reason to use BuildKite, which AFAIU
> > can automatically provision and operate cloud resources for us?
> That's true for buildbot as well [1] including openstack and mesos support which
> would be nice for physical machines.
I think we should try to reduce the number of services we have to
configure and sysadmin, it has a non-zero cost in time and money.
> With the ursa machines down we ended up with two missing services:
> - nightly trigger and report: I've already ported the cron jobs triggering the
> builds and reporting to github actions [2]
> - the comment bot: it is implemented in ursabot including the github event
> listener, so buildkite wouldn't provide a solution here because we need
> to maintain a web service for it.
>
> The solution for the comment bot could be to factor out the implementation
> from ursabot and run it from a github actions build triggered using the
> issue_comment event [3] (buildkite doesn't seem to support it currently [4]).
>
> We can also host the buildbot buildmaster in the cloud, at least until we
> decomission all of its services.
>
> [1] https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers
> [2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows
> [3] https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment
> [4] https://github.com/buildkite/feedback/issues/288
> >
> >
> > Le 04/03/2020 à 16:21, Wes McKinney a écrit :
> > > hi folks,
> > >
> > > The tornado the night before last in Nashville, Tennessee temporarily
> > > disabled the physical hardware that I have been running there where
> > > we've been running "Ursabot" builds and where we've been experimenting
> > > with other self-hosted CI solutions like GitHub Actions Self-Hosted
> > > Runners and Buildkite.
> > >
> > > While dedicated physical hardware can be useful to reduce cloud
> > > computing costs, I think this natural disaster should help inform our
> > > approach to this problem:
> > >
> > > * In the event that physical hosted infrastructure becomes
> > > unavailable, we eventually should have the capability to spin up
> > > machines in the cloud with the desired properties (GCE provides both
> > > Linux and Windows VMs, for example)
> > > * Adding new machines to our CI process ideally should not require a
> > > human in-the-loop (GHA presently requires a human -- in particular
> > > someone from ASF Infra -- in the loop to add workers, so this IMHO
> > > should be taken into consideration)
> > >
> > > Any other thoughts about this topic would be welcome.
> > >
> > > Thanks
> > > Wes
> > >
> > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
> > > <sz...@gmail.com> wrote:
> > >>
> > >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
> > >>>
> > >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
> > >>> <sz...@gmail.com> wrote:
> > >>>>
> > >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> > >>>>>
> > >>>>> hi Ganesh,
> > >>>>>
> > >>>>> Thanks for writing.
> > >>>>>
> > >>>>> I've been working on setting up Buildkite (BK) as a way for third
> > >>>>> parties for attach machines to run builds on, with a free organization
> > >>>>> at
> > >>>>>
> > >>>>> https://buildkite.com/apache-arrow
> > >>>>>
> > >>>>> Configuring a new machine to accept builds is very easy [1] and takes
> > >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work
> > >>>>> on Windows). Currently I've attached 6 machines:
> > >>>>>
> > >>>>> * 2 CUDA-capable Linux x86
> > >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable
> > >>>>> * 1 macOS
> > >>>>>
> > >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds
> > >>>>> triggered in BK can report commit statuses on GitHub [2]
> > >>>>>
> > >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this
> > >>>>> also but the workflow for new machines to be contributed needs to be
> > >>>>> proven out.
> > >>>> I've already tried it out, and setting up self-hosted github runners is just as
> > >>>> easy as with buildkite, drawbacks:
> > >>>
> > >>> I don't mean to be argumentative, but I don't see how this can be true
> > >>> if we don't have access to the "Settings" tab on GitHub
> > >> On a fork where I have access for that tab.
> > >>>
> > >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
> > >>>
> > >>>> - I'm unsure how would the tagging selection work in practice [1]
> > >>>> - We won't have access to the runners dashboard in lack of admin rights
> > >>>> for the apache/arrow repository - so we need to test out the workflow.
> > >>>>
> > >>>> I've created an INFRA ticket to get some information and to track it:
> > >>>> https://issues.apache.org/jira/browse/INFRA-19875
> > >>>>
> > >>>> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Wes
> > >>>>>
> > >>>>> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> > >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217
> > >>>>>
> > >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>> I am following up on the discussion from here
> > >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have
> > >>>>>> dedicated arm hardware for CI setup. We can surely help with that if we get
> > >>>>>> a go-ahead from the project.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ganesh
> > >>>>>>
> > >>>>>> --
> > >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup
on dedicated arm hardware]
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> It sounds like this would be a good reason to use BuildKite, which AFAIU
> can automatically provision and operate cloud resources for us?
That's true for buildbot as well [1] including openstack and mesos support which
would be nice for physical machines.
With the ursa machines down we ended up with two missing services:
- nightly trigger and report: I've already ported the cron jobs triggering the
builds and reporting to github actions [2]
- the comment bot: it is implemented in ursabot including the github event
listener, so buildkite wouldn't provide a solution here because we need
to maintain a web service for it.
The solution for the comment bot could be to factor out the implementation
from ursabot and run it from a github actions build triggered using the
issue_comment event [3] (buildkite doesn't seem to support it currently [4]).
We can also host the buildbot buildmaster in the cloud, at least until we
decomission all of its services.
[1] https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers
[2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows
[3] https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment
[4] https://github.com/buildkite/feedback/issues/288
>
>
> Le 04/03/2020 à 16:21, Wes McKinney a écrit :
> > hi folks,
> >
> > The tornado the night before last in Nashville, Tennessee temporarily
> > disabled the physical hardware that I have been running there where
> > we've been running "Ursabot" builds and where we've been experimenting
> > with other self-hosted CI solutions like GitHub Actions Self-Hosted
> > Runners and Buildkite.
> >
> > While dedicated physical hardware can be useful to reduce cloud
> > computing costs, I think this natural disaster should help inform our
> > approach to this problem:
> >
> > * In the event that physical hosted infrastructure becomes
> > unavailable, we eventually should have the capability to spin up
> > machines in the cloud with the desired properties (GCE provides both
> > Linux and Windows VMs, for example)
> > * Adding new machines to our CI process ideally should not require a
> > human in-the-loop (GHA presently requires a human -- in particular
> > someone from ASF Infra -- in the loop to add workers, so this IMHO
> > should be taken into consideration)
> >
> > Any other thoughts about this topic would be welcome.
> >
> > Thanks
> > Wes
> >
> > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
> > <sz...@gmail.com> wrote:
> >>
> >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
> >>>
> >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
> >>> <sz...@gmail.com> wrote:
> >>>>
> >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> >>>>>
> >>>>> hi Ganesh,
> >>>>>
> >>>>> Thanks for writing.
> >>>>>
> >>>>> I've been working on setting up Buildkite (BK) as a way for third
> >>>>> parties for attach machines to run builds on, with a free organization
> >>>>> at
> >>>>>
> >>>>> https://buildkite.com/apache-arrow
> >>>>>
> >>>>> Configuring a new machine to accept builds is very easy [1] and takes
> >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work
> >>>>> on Windows). Currently I've attached 6 machines:
> >>>>>
> >>>>> * 2 CUDA-capable Linux x86
> >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable
> >>>>> * 1 macOS
> >>>>>
> >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds
> >>>>> triggered in BK can report commit statuses on GitHub [2]
> >>>>>
> >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this
> >>>>> also but the workflow for new machines to be contributed needs to be
> >>>>> proven out.
> >>>> I've already tried it out, and setting up self-hosted github runners is just as
> >>>> easy as with buildkite, drawbacks:
> >>>
> >>> I don't mean to be argumentative, but I don't see how this can be true
> >>> if we don't have access to the "Settings" tab on GitHub
> >> On a fork where I have access for that tab.
> >>>
> >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
> >>>
> >>>> - I'm unsure how would the tagging selection work in practice [1]
> >>>> - We won't have access to the runners dashboard in lack of admin rights
> >>>> for the apache/arrow repository - so we need to test out the workflow.
> >>>>
> >>>> I've created an INFRA ticket to get some information and to track it:
> >>>> https://issues.apache.org/jira/browse/INFRA-19875
> >>>>
> >>>> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> >>>>>
> >>>>> Thanks,
> >>>>> Wes
> >>>>>
> >>>>> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217
> >>>>>
> >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>> I am following up on the discussion from here
> >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have
> >>>>>> dedicated arm hardware for CI setup. We can surely help with that if we get
> >>>>>> a go-ahead from the project.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ganesh
> >>>>>>
> >>>>>> --
> >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup
on dedicated arm hardware]
Posted by Antoine Pitrou <an...@python.org>.
It sounds like this would be a good reason to use BuildKite, which AFAIU
can automatically provision and operate cloud resources for us?
Le 04/03/2020 à 16:21, Wes McKinney a écrit :
> hi folks,
>
> The tornado the night before last in Nashville, Tennessee temporarily
> disabled the physical hardware that I have been running there where
> we've been running "Ursabot" builds and where we've been experimenting
> with other self-hosted CI solutions like GitHub Actions Self-Hosted
> Runners and Buildkite.
>
> While dedicated physical hardware can be useful to reduce cloud
> computing costs, I think this natural disaster should help inform our
> approach to this problem:
>
> * In the event that physical hosted infrastructure becomes
> unavailable, we eventually should have the capability to spin up
> machines in the cloud with the desired properties (GCE provides both
> Linux and Windows VMs, for example)
> * Adding new machines to our CI process ideally should not require a
> human in-the-loop (GHA presently requires a human -- in particular
> someone from ASF Infra -- in the loop to add workers, so this IMHO
> should be taken into consideration)
>
> Any other thoughts about this topic would be welcome.
>
> Thanks
> Wes
>
> On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
>>
>> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
>>>
>>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
>>> <sz...@gmail.com> wrote:
>>>>
>>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
>>>>>
>>>>> hi Ganesh,
>>>>>
>>>>> Thanks for writing.
>>>>>
>>>>> I've been working on setting up Buildkite (BK) as a way for third
>>>>> parties for attach machines to run builds on, with a free organization
>>>>> at
>>>>>
>>>>> https://buildkite.com/apache-arrow
>>>>>
>>>>> Configuring a new machine to accept builds is very easy [1] and takes
>>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work
>>>>> on Windows). Currently I've attached 6 machines:
>>>>>
>>>>> * 2 CUDA-capable Linux x86
>>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable
>>>>> * 1 macOS
>>>>>
>>>>> We're still waiting on ASF Infra to twiddle some bits so that builds
>>>>> triggered in BK can report commit statuses on GitHub [2]
>>>>>
>>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this
>>>>> also but the workflow for new machines to be contributed needs to be
>>>>> proven out.
>>>> I've already tried it out, and setting up self-hosted github runners is just as
>>>> easy as with buildkite, drawbacks:
>>>
>>> I don't mean to be argumentative, but I don't see how this can be true
>>> if we don't have access to the "Settings" tab on GitHub
>> On a fork where I have access for that tab.
>>>
>>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
>>>
>>>> - I'm unsure how would the tagging selection work in practice [1]
>>>> - We won't have access to the runners dashboard in lack of admin rights
>>>> for the apache/arrow repository - so we need to test out the workflow.
>>>>
>>>> I've created an INFRA ticket to get some information and to track it:
>>>> https://issues.apache.org/jira/browse/INFRA-19875
>>>>
>>>> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
>>>>>
>>>>> Thanks,
>>>>> Wes
>>>>>
>>>>> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
>>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217
>>>>>
>>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I am following up on the discussion from here
>>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have
>>>>>> dedicated arm hardware for CI setup. We can surely help with that if we get
>>>>>> a go-ahead from the project.
>>>>>>
>>>>>> Thanks,
>>>>>> Ganesh
>>>>>>
>>>>>> --
>>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Dedicated hardware for Arrow CI / benchmarking [was Re: CI setup on
dedicated arm hardware]
Posted by Wes McKinney <we...@gmail.com>.
hi folks,
The tornado the night before last in Nashville, Tennessee temporarily
disabled the physical hardware that I have been running there where
we've been running "Ursabot" builds and where we've been experimenting
with other self-hosted CI solutions like GitHub Actions Self-Hosted
Runners and Buildkite.
While dedicated physical hardware can be useful to reduce cloud
computing costs, I think this natural disaster should help inform our
approach to this problem:
* In the event that physical hosted infrastructure becomes
unavailable, we eventually should have the capability to spin up
machines in the cloud with the desired properties (GCE provides both
Linux and Windows VMs, for example)
* Adding new machines to our CI process ideally should not require a
human in-the-loop (GHA presently requires a human -- in particular
someone from ASF Infra -- in the loop to add workers, so this IMHO
should be taken into consideration)
Any other thoughts about this topic would be welcome.
Thanks
Wes
On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
> > <sz...@gmail.com> wrote:
> > >
> > > On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > hi Ganesh,
> > > >
> > > > Thanks for writing.
> > > >
> > > > I've been working on setting up Buildkite (BK) as a way for third
> > > > parties for attach machines to run builds on, with a free organization
> > > > at
> > > >
> > > > https://buildkite.com/apache-arrow
> > > >
> > > > Configuring a new machine to accept builds is very easy [1] and takes
> > > > less than 60 seconds on Linux or macOS (though maybe a bit more work
> > > > on Windows). Currently I've attached 6 machines:
> > > >
> > > > * 2 CUDA-capable Linux x86
> > > > * 3 armhf machines (not super high-powered), 1 CUDA-capable
> > > > * 1 macOS
> > > >
> > > > We're still waiting on ASF Infra to twiddle some bits so that builds
> > > > triggered in BK can report commit statuses on GitHub [2]
> > > >
> > > > It's possible we can use self-hosted GitHub Actions (GHA) for this
> > > > also but the workflow for new machines to be contributed needs to be
> > > > proven out.
> > > I've already tried it out, and setting up self-hosted github runners is just as
> > > easy as with buildkite, drawbacks:
> >
> > I don't mean to be argumentative, but I don't see how this can be true
> > if we don't have access to the "Settings" tab on GitHub
> On a fork where I have access for that tab.
> >
> > https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
> >
> > > - I'm unsure how would the tagging selection work in practice [1]
> > > - We won't have access to the runners dashboard in lack of admin rights
> > > for the apache/arrow repository - so we need to test out the workflow.
> > >
> > > I've created an INFRA ticket to get some information and to track it:
> > > https://issues.apache.org/jira/browse/INFRA-19875
> > >
> > > [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> > > >
> > > > Thanks,
> > > > Wes
> > > >
> > > > [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> > > > [2]: https://issues.apache.org/jira/browse/INFRA-19217
> > > >
> > > > On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> > > > >
> > > > > Hi,
> > > > > I am following up on the discussion from here
> > > > > <https://github.com/apache/arrow/pull/6253>, with interest to have
> > > > > dedicated arm hardware for CI setup. We can surely help with that if we get
> > > > > a go-ahead from the project.
> > > > >
> > > > > Thanks,
> > > > > Ganesh
> > > > >
> > > > > --
> > > > > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: CI setup on dedicated arm hardware
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <we...@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
> <sz...@gmail.com> wrote:
> >
> > On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > hi Ganesh,
> > >
> > > Thanks for writing.
> > >
> > > I've been working on setting up Buildkite (BK) as a way for third
> > > parties for attach machines to run builds on, with a free organization
> > > at
> > >
> > > https://buildkite.com/apache-arrow
> > >
> > > Configuring a new machine to accept builds is very easy [1] and takes
> > > less than 60 seconds on Linux or macOS (though maybe a bit more work
> > > on Windows). Currently I've attached 6 machines:
> > >
> > > * 2 CUDA-capable Linux x86
> > > * 3 armhf machines (not super high-powered), 1 CUDA-capable
> > > * 1 macOS
> > >
> > > We're still waiting on ASF Infra to twiddle some bits so that builds
> > > triggered in BK can report commit statuses on GitHub [2]
> > >
> > > It's possible we can use self-hosted GitHub Actions (GHA) for this
> > > also but the workflow for new machines to be contributed needs to be
> > > proven out.
> > I've already tried it out, and setting up self-hosted github runners is just as
> > easy as with buildkite, drawbacks:
>
> I don't mean to be argumentative, but I don't see how this can be true
> if we don't have access to the "Settings" tab on GitHub
On a fork where I have access for that tab.
>
> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
>
> > - I'm unsure how would the tagging selection work in practice [1]
> > - We won't have access to the runners dashboard in lack of admin rights
> > for the apache/arrow repository - so we need to test out the workflow.
> >
> > I've created an INFRA ticket to get some information and to track it:
> > https://issues.apache.org/jira/browse/INFRA-19875
> >
> > [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> > >
> > > Thanks,
> > > Wes
> > >
> > > [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> > > [2]: https://issues.apache.org/jira/browse/INFRA-19217
> > >
> > > On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> > > >
> > > > Hi,
> > > > I am following up on the discussion from here
> > > > <https://github.com/apache/arrow/pull/6253>, with interest to have
> > > > dedicated arm hardware for CI setup. We can surely help with that if we get
> > > > a go-ahead from the project.
> > > >
> > > > Thanks,
> > > > Ganesh
> > > >
> > > > --
> > > > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: CI setup on dedicated arm hardware
Posted by Wes McKinney <we...@gmail.com>.
On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs
<sz...@gmail.com> wrote:
>
> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > hi Ganesh,
> >
> > Thanks for writing.
> >
> > I've been working on setting up Buildkite (BK) as a way for third
> > parties for attach machines to run builds on, with a free organization
> > at
> >
> > https://buildkite.com/apache-arrow
> >
> > Configuring a new machine to accept builds is very easy [1] and takes
> > less than 60 seconds on Linux or macOS (though maybe a bit more work
> > on Windows). Currently I've attached 6 machines:
> >
> > * 2 CUDA-capable Linux x86
> > * 3 armhf machines (not super high-powered), 1 CUDA-capable
> > * 1 macOS
> >
> > We're still waiting on ASF Infra to twiddle some bits so that builds
> > triggered in BK can report commit statuses on GitHub [2]
> >
> > It's possible we can use self-hosted GitHub Actions (GHA) for this
> > also but the workflow for new machines to be contributed needs to be
> > proven out.
> I've already tried it out, and setting up self-hosted github runners is just as
> easy as with buildkite, drawbacks:
I don't mean to be argumentative, but I don't see how this can be true
if we don't have access to the "Settings" tab on GitHub
https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
> - I'm unsure how would the tagging selection work in practice [1]
> - We won't have access to the runners dashboard in lack of admin rights
> for the apache/arrow repository - so we need to test out the workflow.
>
> I've created an INFRA ticket to get some information and to track it:
> https://issues.apache.org/jira/browse/INFRA-19875
>
> [1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
> >
> > Thanks,
> > Wes
> >
> > [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> > [2]: https://issues.apache.org/jira/browse/INFRA-19217
> >
> > On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> > >
> > > Hi,
> > > I am following up on the discussion from here
> > > <https://github.com/apache/arrow/pull/6253>, with interest to have
> > > dedicated arm hardware for CI setup. We can surely help with that if we get
> > > a go-ahead from the project.
> > >
> > > Thanks,
> > > Ganesh
> > >
> > > --
> > > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: CI setup on dedicated arm hardware
Posted by Krisztián Szűcs <sz...@gmail.com>.
On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <we...@gmail.com> wrote:
>
> hi Ganesh,
>
> Thanks for writing.
>
> I've been working on setting up Buildkite (BK) as a way for third
> parties for attach machines to run builds on, with a free organization
> at
>
> https://buildkite.com/apache-arrow
>
> Configuring a new machine to accept builds is very easy [1] and takes
> less than 60 seconds on Linux or macOS (though maybe a bit more work
> on Windows). Currently I've attached 6 machines:
>
> * 2 CUDA-capable Linux x86
> * 3 armhf machines (not super high-powered), 1 CUDA-capable
> * 1 macOS
>
> We're still waiting on ASF Infra to twiddle some bits so that builds
> triggered in BK can report commit statuses on GitHub [2]
>
> It's possible we can use self-hosted GitHub Actions (GHA) for this
> also but the workflow for new machines to be contributed needs to be
> proven out.
I've already tried it out, and setting up self-hosted github runners is just as
easy as with buildkite, drawbacks:
- I'm unsure how would the tagging selection work in practice [1]
- We won't have access to the runners dashboard in lack of admin rights
for the apache/arrow repository - so we need to test out the workflow.
I've created an INFRA ticket to get some information and to track it:
https://issues.apache.org/jira/browse/INFRA-19875
[1] https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner
>
> Thanks,
> Wes
>
> [1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
> [2]: https://issues.apache.org/jira/browse/INFRA-19217
>
> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
> >
> > Hi,
> > I am following up on the discussion from here
> > <https://github.com/apache/arrow/pull/6253>, with interest to have
> > dedicated arm hardware for CI setup. We can surely help with that if we get
> > a go-ahead from the project.
> >
> > Thanks,
> > Ganesh
> >
> > --
> > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
Re: CI setup on dedicated arm hardware
Posted by Wes McKinney <we...@gmail.com>.
hi Ganesh,
Thanks for writing.
I've been working on setting up Buildkite (BK) as a way for third
parties for attach machines to run builds on, with a free organization
at
https://buildkite.com/apache-arrow
Configuring a new machine to accept builds is very easy [1] and takes
less than 60 seconds on Linux or macOS (though maybe a bit more work
on Windows). Currently I've attached 6 machines:
* 2 CUDA-capable Linux x86
* 3 armhf machines (not super high-powered), 1 CUDA-capable
* 1 macOS
We're still waiting on ASF Infra to twiddle some bits so that builds
triggered in BK can report commit statuses on GitHub [2]
It's possible we can use self-hosted GitHub Actions (GHA) for this
also but the workflow for new machines to be contributed needs to be
proven out.
Thanks,
Wes
[1]: https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh
[2]: https://issues.apache.org/jira/browse/INFRA-19217
On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <ga...@linaro.org> wrote:
>
> Hi,
> I am following up on the discussion from here
> <https://github.com/apache/arrow/pull/6253>, with interest to have
> dedicated arm hardware for CI setup. We can surely help with that if we get
> a go-ahead from the project.
>
> Thanks,
> Ganesh
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t