You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "kevinzs2048 (via GitHub)" <gi...@apache.org> on 2024/12/09 06:50:36 UTC
[I] Setup the Arm64 CI [incubator-gluten]
kevinzs2048 opened a new issue, #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181
### Description
Hi Gluten,
I'm from Linaro - an Arm open source organization, usually we work with Arm and other Linaro members to enhance the open source ecosystem support. We want to see if there are the possibility to introduce the Arm64 CI for Gluten. I do see that there are some related jobs, #7654 #7814, and now I can see that there is a Gluten ClickHouse Arm64 CI jobs up and running.
Just want to bring the discussion here for the further to make Arm64 first class citizen for Gluten. I suppose there are more CI jobs should be setup for Arm64 environment, and more CI resources needed, we want to help on this work. If possible, I'd like to know the steps(or essential CI jobs, node resources specifications) are needed to proceed. Look forward to hear from you. And I'd like to join any further discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2915588954
> Should I split the PR and create one for the build and cpp/tpc tests first?
Sure.
> maybe submit it to nightly check and let it failed now, and move to per pr check after fixed?
Maybe, you can 1) postpone adding spark test 2) add arm test job running with Spark UTs, but removing/excluding the failed Spark test (assume only few tests failed caused by the bug you are fixing in upstream velox).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #8181: Setup the Arm64 CI
URL: https://github.com/apache/incubator-gluten/issues/8181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2915384649
@PHILO-HE @zhouyuan I'm thinking after all bugs are fixed, I can raise a PR for both the build and UTs(Spark-3.5/cpp/tpc). But it seems that it takes a longer time than I expected. Should I split the PR and create one for the build and cpp/tpc tests first? leave spark-3.5 uts for future? or maybe submit it to nightly check and let it failed now, and move to per pr check after fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2782319757
> I suggest to firstly add a simply CI job to verify 1) Gluten build on Arm and 2) TPC-H/DS with Spark-3.5 on Arm, which may expose almost all Arm related issues.
>
> I feel it is unnecessary to add a CI job to test Spark-3.x UTs on Arm, since many Spark UT failures are platform independent, which means the existing x86 CI is enough to monitor such failures. As I suggested, maybe we can first add a simple Arm CI and observe some time to see whether we need to expand the test.
>
> cc [@kevinw66](https://github.com/kevinw66)
Actually two bugs are discovered by Spark UTs, while TPC-H/DS seems all correct at the time.
https://github.com/apache/incubator-gluten/issues/8894
https://github.com/facebookincubator/velox/issues/12673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2789256968
Can we have a basic building check for ARM firstly? This would not introduce high cost in the apache github runner side.
related: https://github.com/apache/incubator-gluten/pull/8982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2782726230
@PHILO-HE
> For #12673, does it also happen on x86 when avx2 is disabled? If yes, it is more like that we lacked test on platform without avx2, not an issue only occurs on Arm.
I think it does, I didn't disable the avx2, just delete related code.
> My point is it seems unnecessary to add another CI job to run entire Spark UTs on ARM for each PR.
Although it's said so, these bugs were indeed discovered through Spark UTs. If we disable it, some problems may become more hidden. Currently, the CIs I've prepared for Arm are as shown in the following figure. I wonder if the resources have been exceeded? What about keeping only one version of Spark for Arm（Remove Spark34/Spark35-jdk17/Spark35-scala213 and two tpc tests on centos8）？In that case, the number of newly added jobs is 9.
<img width="322" alt="Image" src="https://github.com/user-attachments/assets/5452d661-4023-46e0-99c9-0c6e80db1668" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2785801663
@kevinw66, actually we are inspecting the test redundancy in CI and we have removed some test jobs. Our considerations are not just for reducing the resources usage to avoid hitting the limit, but also for making pr validation more efficient. It's encouraged to add necessary tests to improve Gluten robustness, but we should avoid too much redundancy. Thus, it can be possible that we are able to add other necessary tests in the future.
If you still think it's quite necessary to run Spark-3.2 UTs for Arm validation, one thought is you can add a nightly or weekly CI job, which is not in time to expose issues, but may be acceptable if failure rarely happens. In addition, you can add a simple CI job for per pr validation as I mentioned before. Another thought is you may directly change to use Arm for the current Spark-3.2 UTs job, assuming other Spark test jobs may be sufficient for x86 validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "kevinw66 (via GitHub)" <gi...@apache.org>.
kevinw66 commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2918400395
> > Should I split the PR and create one for the build and cpp/tpc tests first?
>
> Sure.
>
> > maybe submit it to nightly check and let it failed now, and move to per pr check after fixed?
>
> Maybe, you can 1) postpone adding spark test on arm or 2) add spark test for per pr validation, but removing/excluding the failed Spark test (assume only few tests failed caused by the bug you are fixing in upstream velox).
Thanks, I'll postpone the spark uts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2782182474
I suggest to firstly add a simply CI job to verify 1) Gluten build on Arm and 2) TPC-H/DS with Spark-3.5 on Arm, which may expose almost all Arm related issues.
I feel it is unnecessary to add a CI job to test Spark-3.x UTs on Arm, since many Spark UT failures are platform independent, which means the existing x86 CI is enough to monitor such failures. As I suggested, maybe we can first add a simple Arm CI and observe some time to see whether we need to expand the test.
cc @kevinw66
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Setup the Arm64 CI [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on issue #8181:
URL: https://github.com/apache/incubator-gluten/issues/8181#issuecomment-2782355807
@kevinw66, #8894 is tpch test with some spark config enabled, not Spark's own UT. It's put in VeloxScanSuite.scala in backends-velox module. If necessary, we can enable these tests under this module for Arm.
For #12673, does it also happen on x86 when avx2 is disabled? If yes, it is more like that we lacked test on platform without avx2, not an issue only occurs on Arm.
My point is it seems unnecessary to add another CI job to run entire Spark UTs on ARM for each PR. If really required, it can be moved to nightly or weekly CI. But firstly, I recommend to add a simple CI for per pr validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org