You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/03 03:24:10 UTC
[GitHub] [spark] Yikun opened a new pull request #35088: [SPARK-37758][PYTHON][BUILD] Enable PySpark test scheduled job on ARM runner
Yikun opened a new pull request #35088:
URL: https://github.com/apache/spark/pull/35088
### What changes were proposed in this pull request?
This patch adds the pyspark arm scheduled job on ARM runner.
- Add pyspark-arm64-scheduled job.
- Runs on arm64 self-hosted runner with `ubuntu-20.04-arm64` label
- Using `yikunkero/apache-spark-github-action-image:arm64` in arm test.
- Installed minicoda with `-u` (upgrade if exist otherwise install directly), and also add multi-arch improvement.
### Why are the changes needed?
Migrate PySpark Arm Job from Jenkins to GitHub Actions.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Trigger local test based on this patch in: https://github.com/Yikun/spark/pull/53
- CI passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #35088: [SPARK-37758][PYTHON][BUILD] Enable PySpark test scheduled job on ARM runner
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #35088:
URL: https://github.com/apache/spark/pull/35088#issuecomment-1080097843
Just out of curious: Is this still work in progress ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #35088: [SPARK-37758][PYTHON][BUILD] Enable PySpark test scheduled job on ARM runner
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #35088:
URL: https://github.com/apache/spark/pull/35088#issuecomment-1080131756
@itholic Thanks for your attention, I've been busy with other things lately. The support of this arm64, there is no progress for the time being, but I will continue this work.
The purpose of this PR is to (same with https://github.com/apache/spark/pull/35049):
1. Find some problems of Spark on arm in time, and report them to the community quickly.
2. Provide an open mechanism that allows Spark developers to quickly verify on arm: Just like [some previous PR](https://github.com/apache/spark/pull/35078#issuecomment-1004516832) did by me manually.
I've tried a few things ([support pyspark arm64 ci images](https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/pull/6), [self-hosted demo](https://lists.apache.org/thread/mq121fmggs9bbo5ll4ld1qqjv7cshldp)), but there seem to be some diff suggestions in the community.
So I temporarily use the downstream repo to ensure the verification of spark on arm linux:
https://github.com/spark-arm/spark/actions/workflows/build_and_test.yml
(However, tasks often fail due to conflicts with upstream, I have to sync manaully when I get time)
What's the next step:
1. Backport some useful configurations to upstream spark and using workflow_run way to make configuration separately to reduce merge conflict.
2. Add workflow trigger on demand in downstream.
Of course, I also very much hope that it can be truly integrated in the upstream community, thereby reducing downstream costs and ensuring that spark on arm64 passed always. feel free to ping me if you have any other suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35088: [SPARK-37758][PYTHON][BUILD] Enable PySpark test scheduled job on ARM runner
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35088:
URL: https://github.com/apache/spark/pull/35088#issuecomment-1080131756
Thanks for your attention, I've been busy with other things lately. The support of this arm64, there is no progress for the time being, but I will continue this work.
The purpose of this PR is to (same with https://github.com/apache/spark/pull/35049):
1. Find some problems of Spark on arm in time, and report them to the community quickly.
2. Provide an open mechanism that allows Spark developers to quickly verify on arm: Just like [some previous PR](https://github.com/apache/spark/pull/35078#issuecomment-1004516832) did by me manually.
I've tried a few things ([support pyspark arm64 ci images](https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/pull/6), [self-hosted demo](https://lists.apache.org/thread/mq121fmggs9bbo5ll4ld1qqjv7cshldp)), but there seem to be some diff suggestions in the community.
So I temporarily use the downstream repo to ensure the verification of spark on arm linux:
https://github.com/spark-arm/spark/actions/workflows/build_and_test.yml
(However, tasks often fail due to conflicts with upstream.)
What's the next step:
1. Backport some useful configurations to upstream spark and using workflow_run way to make configuration separately to reduce merge conflict.
2. Add workflow trigger on demand in downstream.
Of course, I also very much hope that it can be truly integrated in the upstream community, thereby reducing downstream costs and ensuring that spark on arm64 passed always. feel free to ping me if you have any other suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org