You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/08 09:11:35 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
HyukjinKwon opened a new pull request #32092:
URL: https://github.com/apache/spark/pull/32092
### What changes were proposed in this pull request?
This PR proposes to leverage the GitHub Actions resources from the forked repositories instead of sharing the resources in ASF.
This is how it works:
1. "Build and test" (`build_and_test.yml`) triggers a build on any commit on any branch (except `branch-*.*`), which means:
- The original repository will trigger the build on any commits in `master` branch
- The forked repository will trigger the build on any commit in any branch.
2. The build triggered in the forked repository will checkout the original repository's `master` branch locally, and merge the branch from the forked repository into the original repository's `master` branch locally.
Therefore, the tests in the forked repository will run after sync'ed with the original repository's `master` branch.
3. In the original repository, it triggers a workflow that detects the workflow triggered in the forked repository, and add a comment on the PR, pointing out the workflow.
In short,
1. You create a PR, your repository triggers the workflow. It uses the resources allocated to you.
2. Apache Spark repository finds your workflow, and add it as a comment in the PR
### Why are the changes needed?
ASF shares the resources across all the ASF projects, which makes the development slow down.
Please see also:
- Discussion in the builds@a.o mailing list: https://lists.apache.org/x/thread.html/r48d079eeff292254db22705c8ef8618f87ff7adc68d56c4e5d0b4105@%3Cbuilds.apache.org%3E
- Infra ticket: https://issues.apache.org/jira/browse/INFRA-21646
By distributing the workflows to use author's resources, we can get around this issue.
### Does this PR introduce _any_ user-facing change?
No, this is a dev-only change.
### How was this patch tested?
Manually tested at https://github.com/HyukjinKwon/spark/pull/34 and https://github.com/HyukjinKwon/spark/pulls/33.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-816327236
Thanks @potiuk. Yeah, I think it makes sense to do that seamlessly with updating the status checks too. Borrowing the idea from cancel workflow is a good idea. If we're going this way, I will take a separate look for that.
FWIW, I do think this is still a workaround that disables GitHub Actions to work out of the box. In the mailing list, the permanent solution is being discussed, which I think makes sense and reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609754238
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -3,15 +3,8 @@ name: Build and test
on:
push:
branches:
- - master
- pull_request:
- branches:
- - master
- workflow_dispatch:
- inputs:
- target:
- description: 'Target branch to run'
- required: true
+ - '**'
+ - '!branch-*.*'
Review comment:
why exclude `'!branch-*.*'`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609758587
##########
File path: .github/workflows/notify_test_workflow.yml
##########
@@ -0,0 +1,40 @@
+name: Notify test workflow
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+
+jobs:
+ notify:
+ runs-on: ubuntu-20.04
+ steps:
+ - name: "Notify test workflow"
+ uses: actions/github-script@v3
+ if: ${{ github.base_ref == 'master' }}
Review comment:
I haven't read the code yet, but backport PRs are not common and I think it's OK to use the ASF resource to run tests for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk edited a comment on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818023289
@tgravescs : Github Actions and storage are FREE for public repositories. If you look for the smaller footprint in the pricing page (bold is mine):
- Unlimited public/private repositories
- Unlimited collaborators
- Included2,000 Actions minutes/month
- **Free for public repositories**
- Included500MB of GitHub Packages storage
- **Free for public repositories**
- IncludedCommunity Support
There are some limitations when it comes to number of runners available (so if you have more builds they will queue in your personal queue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815917941
It's great. Thank you, @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815636347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815738202
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-816005282
This is an excellent idea ! I :heart: it. Are all the pieces already working :) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815597047
**[Test build #137066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137066/testReport)** for PR 32092 at commit [`6dabe60`](https://github.com/apache/spark/commit/6dabe60b04ecbef71f59cc5b5650d08ec2dcebae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609755629
##########
File path: .github/workflows/notify_test_workflow.yml
##########
@@ -0,0 +1,40 @@
+name: Notify test workflow
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+
+jobs:
+ notify:
+ runs-on: ubuntu-20.04
+ steps:
+ - name: "Notify test workflow"
+ uses: actions/github-script@v3
+ if: ${{ github.base_ref == 'master' }}
Review comment:
What happens for the backport PR to branch-3.0/branch-3.1? Shall we just run the test in Apache/Spark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818023289
@tgravescs : Github Actions and storage are FREE for public repositories. If you look for the smaller footprint in the pricing page (bold is mine):
- Unlimited public/private repositories
- Unlimited collaborators
- Included2,000 Actions minutes/month
- **Free for public repositories**
- Included500MB of GitHub Packages storage
- **Free for public repositories**
- IncludedCommunity Support
There are some limitations when it comes to number of runners available,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818519799
**[Test build #137286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137286/testReport)** for PR 32092 at commit [`74a7674`](https://github.com/apache/spark/commit/74a767475965928e4d95af04e500532534597879).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815638114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41644/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815638114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41644/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815845639
I was surprised that there is such a solution for the resource-shortage issue, nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-816015860
One thing that is missing is how to get back the information about the status of the build in the forked repository. I do not think with the permissions model of GH it is possible to do directly, but I can think of another 'scheduled' workflow that could run in the original repository that could read status from all the PRs that had "forked" builds running.
That would be quite possible. We could even try to utilize "GitHub Check" for that https://docs.github.com/en/rest/reference/checks - I am already using it in Airflow to notify status betwen workflows (I have a 'build image` workflow that notifies the PR that runs a CI workflow this way.
This looks like the regular "yellow/green/red" build status indicator, and it impacts the "mergable" status of the PR.
This way we could have a complete solution, when the PR would become "green" when the forked build succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818632725
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137286/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818605976
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815597047
**[Test build #137066 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137066/testReport)** for PR 32092 at commit [`6dabe60`](https://github.com/apache/spark/commit/6dabe60b04ecbef71f59cc5b5650d08ec2dcebae).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815597385
cc @dongjoon-hyun, @MaxGekk, @maropu, @srowen, @gengliangwang FYI
cc @potiuk too FYI who might be interested in this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-817891924
Definitely seems like a possible way to distribute the load and cost.
Are there limitations on the personal repo GitHub actions?
looking at the free pricing for instance: (https://github.com/pricing)
> 2,000 Actions minutes/month
> 500MB of Packages storage
How many action minutes would an average spark pr take? the test build took 1 hours 27 minutes duration but is that action minutes (https://github.com/MaxGekk/spark/actions/runs/728916852) or do you have to add up each of the individual build modules, java 11, etc?
Similar with storage are log files still available, how long or is there any others differences to deal with?
just wondering some of these because say a user runs out of action minutes or there are problems, then what do we do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-816333000
Agree it's a nasty hack and should be out-of-the-box built in GitHub.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818519799
**[Test build #137286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137286/testReport)** for PR 32092 at commit [`74a7674`](https://github.com/apache/spark/commit/74a767475965928e4d95af04e500532534597879).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r612927283
##########
File path: dev/run-tests.py
##########
@@ -656,16 +656,10 @@ def main():
# If we're running the tests in GitHub Actions, attempt to detect and test
# only the affected modules.
if test_env == "github_actions":
- if os.environ["GITHUB_INPUT_BRANCH"] != "":
Review comment:
@maropu FYI. I think we should update https://spark.apache.org/developer-tools.html because now we always run the tests on each commit in each branch in forked repositories.. I will take a look and fix it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818518657
This is ready for a review. I will take a look for https://github.com/apache/spark/pull/32092#issuecomment-816015860 separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609774874
##########
File path: .github/workflows/notify_test_workflow.yml
##########
@@ -0,0 +1,40 @@
+name: Notify test workflow
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+
+jobs:
+ notify:
+ runs-on: ubuntu-20.04
+ steps:
+ - name: "Notify test workflow"
+ uses: actions/github-script@v3
+ if: ${{ github.base_ref == 'master' }}
Review comment:
Yeah, let's keep it simple for now. We will need to update the file `.github/workflows/build_and_test.yml` in branch-3.2. We can try finding a way to make it work for every new release branch later. (Simply manually update is also fine)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815697398
**[Test build #137066 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137066/testReport)** for PR 32092 at commit [`6dabe60`](https://github.com/apache/spark/commit/6dabe60b04ecbef71f59cc5b5650d08ec2dcebae).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818632725
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137286/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-817605743
I did not have time to take a look yet, and during the week I have a bit limited time but in a week or so I hope to have some POC with the checks in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818631180
**[Test build #137286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137286/testReport)** for PR 32092 at commit [`74a7674`](https://github.com/apache/spark/commit/74a767475965928e4d95af04e500532534597879).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-816016901
> One thing that is missing is how to get back the information about the status of the build in the forked repository. I do not think with the permissions model of GH it is possible to do directly, but I can think of another 'scheduled' workflow that could run in the original repository that could read status from all the PRs that had "forked" builds running.
I think it needs a bit complex logic, but I could possibly write a custom GitHub Action for that similar to what I've done with https://github.com/potiuk/cancel-workflow-runs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32092:
URL: https://github.com/apache/spark/pull/32092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-819189675
Thanks @dongjoon-hyun!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-819214998
Thanks @gengliangwang! (I just fixed a indentation nit)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609754238
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -3,15 +3,8 @@ name: Build and test
on:
push:
branches:
- - master
- pull_request:
- branches:
- - master
- workflow_dispatch:
- inputs:
- target:
- description: 'Target branch to run'
- required: true
+ - '**'
+ - '!branch-*.*'
Review comment:
why exclude `'!branch-*.*'`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818605931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815849462
@HyukjinKwon Thanks for the great effort in testing infra!
+1 for this proposal. I just checked https://github.com/pricing and found that GitHub action is totally free for public repositories. Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-815738202
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137066/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] potiuk edited a comment on pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818023289
@tgravescs : Github Actions and storage are FREE for public repositories. If you look for the smaller footprint in the pricing page (bold is mine):
- Unlimited public/private repositories
- Unlimited collaborators
- Included 2,000 Actions minutes/month
**Free for public repositories**
- Included 500MB of GitHub Packages storage
**Free for public repositories**
- IncludedCommunity Support
There are some limitations when it comes to number of runners available (so if you have more builds they will queue in your personal queue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-818605976
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32092:
URL: https://github.com/apache/spark/pull/32092#issuecomment-819218687
I am merging it into master.
It's a daytime in my time .. so would likely be able to track it if anything goes wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #32092: [DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r609755629
##########
File path: .github/workflows/notify_test_workflow.yml
##########
@@ -0,0 +1,40 @@
+name: Notify test workflow
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+
+jobs:
+ notify:
+ runs-on: ubuntu-20.04
+ steps:
+ - name: "Notify test workflow"
+ uses: actions/github-script@v3
+ if: ${{ github.base_ref == 'master' }}
Review comment:
What happens for the backport PR to branch-3.0/branch-3.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32092: [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32092:
URL: https://github.com/apache/spark/pull/32092#discussion_r612923962
##########
File path: .github/workflows/build_and_test.yml
##########
@@ -3,15 +3,8 @@ name: Build and test
on:
push:
branches:
- - master
- pull_request:
- branches:
- - master
- workflow_dispatch:
- inputs:
- target:
- description: 'Target branch to run'
- required: true
+ - '**'
+ - '!branch-*.*'
Review comment:
```suggestion
- '**'
- '!branch-*.*'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org