You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/28 03:09:29 UTC
[GitHub] [spark] Yikun edited a comment on pull request #35088: [SPARK-37758][PYTHON][BUILD] Enable PySpark test scheduled job on ARM runner
Yikun edited a comment on pull request #35088:
URL: https://github.com/apache/spark/pull/35088#issuecomment-1080131756
@itholic Thanks for your attention, I've been busy with other things lately. The support of this arm64, there is no progress for the time being, but I will continue this work.
The purpose of this PR is to (same with https://github.com/apache/spark/pull/35049):
1. Find some problems of Spark on arm in time, and report them to the community quickly.
2. Provide an open mechanism that allows Spark developers to quickly verify on arm: Just like [some previous PR](https://github.com/apache/spark/pull/35078#issuecomment-1004516832) did by me manually.
I've tried a few things ([support pyspark arm64 ci images](https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/pull/6), [self-hosted demo](https://lists.apache.org/thread/mq121fmggs9bbo5ll4ld1qqjv7cshldp)), but there seem to be some diff suggestions in the community.
So I temporarily use the downstream repo to ensure the verification of spark on arm linux:
https://github.com/spark-arm/spark/actions/workflows/build_and_test.yml
(However, tasks often fail due to conflicts with upstream, I have to sync manaully when I get time)
What's the next step:
1. Backport some useful configurations to upstream spark and using workflow_run way to make configuration separately to reduce merge conflict.
2. Add workflow trigger on demand in downstream.
Of course, I also very much hope that it can be truly integrated in the upstream community, thereby reducing downstream costs and ensuring that spark on arm64 passed always. feel free to ping me if you have any other suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org