You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Sutou Kouhei <ko...@clear-code.com> on 2020/01/27 01:17:31 UTC
ARM CI by ursa-labs/crossbow
Hi,
We're building .deb/.rpm packages for ARM by using QEMU on
x86-64 machine:
https://github.com/apache/arrow/blob/master/dev/tasks/linux-packages/azure.linux.arm64.yml#L36-L50
It's slow. Some jobs can't finish in 6h, Azure Pipelines
build time limit. You can see some *-aarch64 and *-arm64
jobs are failed in
https://github.com/apache/arrow/pull/6276#issuecomment-578006232 .
(Especially building gRPC for Flight is slow.)
It means that we can't build some .deb/.rpm packages for ARM
by Crossbow.
I can build them on my local machine for 0.16.0. Or we can
just ignore these packages because we didn't support
.deb/.rpm for ARM until 0.15.1. .deb/.rpm for ARM support is
a new feature of 0.16.0. This is not a regression. So this
is not a blocker of 0.16.0.
I submitted a pull request to add support for GitHub Actions
to Crossbow: https://github.com/apache/arrow/pull/6286
If we can use GitHub Actions by Crossbow, we can improve ARM
package build performance by adding self-hosted GitHub
Runner for ARM.
See the following self-hosted GitHub Runner:
https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners
It seems that Ursa Labs has ARM machine:
https://github.com/ursa-labs/ursabot/blob/master/workers.yaml#L113
Could you add the ARM machine to
https://github.com/ursa-labs/crossbow/ ? Then we can use
the ARM machine to build .deb/.rpm packages for ARM by
Crossbow.
Thanks,
--
kou
Re: ARM CI by ursa-labs/crossbow
Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi Wes,
Thanks!!!
--
kou
In <CA...@mail.gmail.com>
"Re: ARM CI by ursa-labs/crossbow" on Sun, 26 Jan 2020 23:47:15 -0600,
Wes McKinney <we...@gmail.com> wrote:
> hi Kou,
>
> Sounds like a good idea. We have a couple ARM machines that we can try
> connecting to GHA
>
> * Jetson TX2 dev kit
> * Rockpro64
>
> I don't know that I can personally spend much time on this this week
> but I'll make sure that the machines are up on the network and able to
> be SSH'd into by other Ursa Labs folks
>
> Thanks
>
> On Sun, Jan 26, 2020 at 7:17 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>>
>> Hi,
>>
>> We're building .deb/.rpm packages for ARM by using QEMU on
>> x86-64 machine:
>> https://github.com/apache/arrow/blob/master/dev/tasks/linux-packages/azure.linux.arm64.yml#L36-L50
>>
>> It's slow. Some jobs can't finish in 6h, Azure Pipelines
>> build time limit. You can see some *-aarch64 and *-arm64
>> jobs are failed in
>> https://github.com/apache/arrow/pull/6276#issuecomment-578006232 .
>>
>> (Especially building gRPC for Flight is slow.)
>>
>> It means that we can't build some .deb/.rpm packages for ARM
>> by Crossbow.
>>
>> I can build them on my local machine for 0.16.0. Or we can
>> just ignore these packages because we didn't support
>> .deb/.rpm for ARM until 0.15.1. .deb/.rpm for ARM support is
>> a new feature of 0.16.0. This is not a regression. So this
>> is not a blocker of 0.16.0.
>>
>>
>> I submitted a pull request to add support for GitHub Actions
>> to Crossbow: https://github.com/apache/arrow/pull/6286
>>
>> If we can use GitHub Actions by Crossbow, we can improve ARM
>> package build performance by adding self-hosted GitHub
>> Runner for ARM.
>>
>> See the following self-hosted GitHub Runner:
>> https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners
>>
>>
>> It seems that Ursa Labs has ARM machine:
>> https://github.com/ursa-labs/ursabot/blob/master/workers.yaml#L113
>>
>>
>> Could you add the ARM machine to
>> https://github.com/ursa-labs/crossbow/ ? Then we can use
>> the ARM machine to build .deb/.rpm packages for ARM by
>> Crossbow.
>>
>>
>> Thanks,
>> --
>> kou
Re: ARM CI by ursa-labs/crossbow
Posted by Wes McKinney <we...@gmail.com>.
hi Kou,
Sounds like a good idea. We have a couple ARM machines that we can try
connecting to GHA
* Jetson TX2 dev kit
* Rockpro64
I don't know that I can personally spend much time on this this week
but I'll make sure that the machines are up on the network and able to
be SSH'd into by other Ursa Labs folks
Thanks
On Sun, Jan 26, 2020 at 7:17 PM Sutou Kouhei <ko...@clear-code.com> wrote:
>
> Hi,
>
> We're building .deb/.rpm packages for ARM by using QEMU on
> x86-64 machine:
> https://github.com/apache/arrow/blob/master/dev/tasks/linux-packages/azure.linux.arm64.yml#L36-L50
>
> It's slow. Some jobs can't finish in 6h, Azure Pipelines
> build time limit. You can see some *-aarch64 and *-arm64
> jobs are failed in
> https://github.com/apache/arrow/pull/6276#issuecomment-578006232 .
>
> (Especially building gRPC for Flight is slow.)
>
> It means that we can't build some .deb/.rpm packages for ARM
> by Crossbow.
>
> I can build them on my local machine for 0.16.0. Or we can
> just ignore these packages because we didn't support
> .deb/.rpm for ARM until 0.15.1. .deb/.rpm for ARM support is
> a new feature of 0.16.0. This is not a regression. So this
> is not a blocker of 0.16.0.
>
>
> I submitted a pull request to add support for GitHub Actions
> to Crossbow: https://github.com/apache/arrow/pull/6286
>
> If we can use GitHub Actions by Crossbow, we can improve ARM
> package build performance by adding self-hosted GitHub
> Runner for ARM.
>
> See the following self-hosted GitHub Runner:
> https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners
>
>
> It seems that Ursa Labs has ARM machine:
> https://github.com/ursa-labs/ursabot/blob/master/workers.yaml#L113
>
>
> Could you add the ARM machine to
> https://github.com/ursa-labs/crossbow/ ? Then we can use
> the ARM machine to build .deb/.rpm packages for ARM by
> Crossbow.
>
>
> Thanks,
> --
> kou