You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/01 14:17:51 UTC
[GitHub] [beam] TobKed opened a new pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
TobKed opened a new pull request #11877:
URL: https://github.com/apache/beam/pull/11877
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-636900185
related to [BEAM-9388 - Consider using github actions for building python wheels and more (aka. Transition from Travis)](https://issues.apache.org/jira/browse/BEAM-9388)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-638978380
> How can I preview the action on the fork: [TobKed#3](https://github.com/TobKed/beam/pull/3) ?
@aaltay https://github.com/TobKed/beam/actions?query=branch%3Agithub-actions-build-wheels-linux-macos
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-639190586
> > How can I preview the action on the fork: [TobKed#3](https://github.com/TobKed/beam/pull/3) ?
>
> @aaltay https://github.com/TobKed/beam/actions?query=branch%3Agithub-actions-build-wheels-linux-macos
Super nice! I have not reviewed the steps, I think PR is better for doing the most initial review. We can do a final quick review of this once we settle on this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-641482606
This looks nice. I have a few clarifiying questions:
on pull_request: This is good. Would it trigger on every pull request. This may not be needed. I am not sure what GH resource we will have and what kind of queue there will be. We would not want ot add test load to all PRs.
Questions:
- Can we trigger this with a phrase? And/Or limit it to python changes?
- What is gh-action artifacts? How does one access this to get the artifacts?
on push:
Question:
- What is release-candidate branches? It is release-* ? (If yes, this looks good.)
on schedule: this looks good.
Other questions:
- How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)
> Nightly is addressed by on schedule trigger.
+1.
> I investigated triggering builds manually by using repository_dispatch and curl however build triggered by it is executed on master branch and I think there is no convenient way to run it on different branches. If your question is related to "Build Release Candidate" phase it may be solved here as well since build will be triggered during pushing to master/release/release-candidate branches (and related to it by commit hash).
Yes, I think this works. My question was how could a release manager build wheels. Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-651366970
Nice, thank you! 🎉
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435630823
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
Very nice.
2 follow up questions:
1. Can we add a very last stage that can run: gsutil ls "gs://***/$***GITHUB_REF##*/***/*" -> so that we can get the whole output of the gcs folder all at once.
(btw, do we have a mechanism to clean up these gcs locations?)
2. What is the difference between "Build python wheels / Build wheels on ..." job executing "Upload wheels" step and "Upload wheels to GCS bucket ..." job?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r442594989
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
Sorry, I missed this earlier.
> I think it would be much cleaner and explicit. Did I understand correctly your idea?
Yes. Your understanding is correct. This sounds good to me.
> Clean up:
Both options are good. Can we do both?
> What do you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as artifacts" ?
Sounds good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r434779681
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
Review comment:
Yes, let's add py3.8. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435192970
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
Review comment:
I found information in the documentation ([source](https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources)) which says:
> Note: The Ubuntu 20.04 virtual environment is currently provided as a preview only. The ubuntu-latest YAML workflow label still uses the Ubuntu 18.04 virtual environment.
In this case using `ubuntu-latest` label may be a good idea. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435407993
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
Review comment:
Good point 👍
I've tested and for `build_source` job only `python3 -m pip install -r build-requirements.txt` is required, then `python -m pip install cython` is executed in `build_wheels`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-641137700
After rethinking gh-actions I made some updates which changes CI behavior dependent on the triggering event as presented below:
* **on pull_request** - operates on PR branch, builds wheels and uploads them only to gh-action artifacts ([example-run](https://github.com/TobKed/beam/actions/runs/129159523))
* **on push** - operates on master/release/release-candidate branches, build wheels and uploads them gh-action artifacts and GCS bucket (folder - branch name) ([example-run](https://github.com/TobKed/beam/actions/runs/129158855))
* **on schedule** - operates master branch, builds wheels and uploads them to gh-action artifacts and GCS bucket (folder - branch name), tags last commit on master as `nightly-master` ([example-run](https://github.com/TobKed/beam/actions/runs/129288105))
Additionally I've added `Cancel Previous Runs` workflow (`.gihub/workflows/cancel.yml`) which cancels previous runs on the same branch.
It was inspired by Apache Airflow solution ([link to PR](https://github.com/apache/airflow/pull/9050))
@aaltay answering your questions:
> Could we make it such that:
>
> * we have a cron job that builds nightly (like a snapshot release)
> * and a way to manually trigger this from the release branch, so that the release manager can build wheel files on demand.
1. Nightly is addressed by `on schedule` trigger.
2. I investigated triggering builds manually by using `repository_dispatch` and curl however build triggered by it is executed on master branch and I think there is no convenient way to run it on different branches. If your question is related to "Build Release Candidate" phase it may be solved here as well since build will be triggered during pushing to master/release/release-candidate branches (and related to it by commit hash).
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed removed a comment on pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
Posted by GitBox <gi...@apache.org>.
TobKed removed a comment on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-636900185
related to [BEAM-9388 - Consider using github actions for building python wheels and more (aka. Transition from Travis)](https://issues.apache.org/jira/browse/BEAM-9388)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-638395828
Could we make it such that:
- we have a cron job that builds nightly (like a snapshot release)
- and a way to manually trigger this from the release branch, so that the release manager can build wheel files on demand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r434724157
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
What does this do? Is it re-compressing the source folder. I wonder if we can use the sdist output as it?
(Ideally the resulting GCS output look something close enoguh to a release output, e.g. https://dist.apache.org/repos/dist/release/beam/2.21.0/python/ )
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
Review comment:
@tvalentyn - is py3.8 ready to be added here?
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
Review comment:
Are we planning to add windows here or to another file?
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
Review comment:
I am not familiar with github ci. This is a question for my learning. Does each build step (job?) work in a different os environment?
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
Review comment:
Should we use ubuntu-20.04 instead?
(Apparently this is now supported: https://github.com/actions/virtual-environments/issues/228)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436068238
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
Review comment:
Hopefully yes 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-639667163
cc @potiuk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r433759180
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,91 @@
+on: [push]
+name: Build python wheels
+
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v1
Review comment:
Yes, we should :) thanks for pointing it out, I missed it.
GitHub Actions documentations says [here](https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#using-the-default-python-version):
> We recommend using setup-python to configure the version of Python used in your workflows because it helps make your dependencies explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-636902595
github-actions are currently running on PR in fork: https://github.com/TobKed/beam/pull/3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435190449
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
Review comment:
> You can specify the runner type for each job in a workflow. Each job in a workflow executes in a fresh instance of the virtual machine. All steps in the job execute in the same instance of the virtual machine, allowing the actions in that job to share information using the filesystem.
source: https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436067638
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
1. Currently two steps: `List sources on GCS bucket` and `Copy wheels to GCS bucket` are listing files of specific types. Instead of this two separate steps I could create job which will list all files in specific gcs folder. I think it would be much cleaner and explicit. Did I understand correctly your idea?
About cleaning up these GCS locations I consider two options:
- setting lifecycle management on the bucket which will delete files older than some arbitrary age, e.g. 365 days. I think advantage of this is that will be maintenance free.
- creating another scheduled workflow on github actions which will delete gcs folders if corresponding branch does not exist anymore. Could be scheduled to run e.g. once pre week.
Which option has more sense for you?
2. "Upload" steps perform file upload as artifacts so they could be passed between jobs and being available for download for 90 days (if not deleted earlier). These artifacts are picked up later by "Upload to GCS" jobs. What do you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as artifacts" ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-641886596
@aaltay thank you for review. Answers for your questions:
> on pull_request:
> Can we trigger this with a phrase? And/Or limit it to python changes?
I added `paths` for filtering so only meaningful changes will trigger build check.
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths.
> What is gh-action artifacts? How does one access this to get the artifacts?
GitHub Action artifacts allow to persist data on storage space to share that data with another job.
They can be downloaded from GitHub UI for by API. GitHub stores artifacts for 90 days
You can test it on example run: https://github.com/TobKed/beam/actions/runs/129158855
[GitHub Actions Artifacts documentation](https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#about-workflow-artifacts)
[GitHub API - Download an artifact](https://developer.github.com/v3/actions/artifacts/#download-an-artifact)
> How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)
It is done automatically in `.gihub/workflows/cancel.yml` with usage of https://github.com/styfle/cancel-workflow-action.
This action is checking is there any other previously triggered and running workflow on given branch and cancel them.
It saves resources and prevents colliding actions on GCS bucket.
> What is release-candidate branches? It is release-* ? (If yes, this looks good.)
I think I made mistake here. To be precise `release-*` branch is called `RELEASE_BRANCH` (based on [build_release_candidate.sh#L57](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L57)) and based on [build_release_candidate.sh#L110](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L110) proper pattern for release-candidate branches should be: `v*` (previously this pattern was for `tags` only)
> Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?
Currently source files are build here [build_release_candidate.sh#L174](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L174) based on RELEASE_BRANCH, the release branch is created few lines above [build_release_candidate.sh#L109](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L109), this push will trigger build of python sources and wheels.
Build can be triggered in both ways: by pushing to branches matching patterns or creating PRs to these branches (however due to path filtering build on pull request may be not triggered).
I hope it answers your questions :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] potiuk commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-649336627
:tada:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435631141
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
Review comment:
Perfect. (just to clarify my understanding: we will have a single github ci config, this is correct, right?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435404202
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
Review comment:
Sure, I will add it 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-638238507
cc @kamilwu @damgad @aaltay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed edited a comment on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-638238507
cc @kamilwu @damgad @aaltay @tysonjh
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-651242128
Thank you! Could you check that new workflows work as expected?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435203092
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
It is uploading sdist output (`zip`,`tag.gz`) files as artifacts where they are picked up by `upload_source_to_gcs` jobs. An additional advantage of the artifacts is fact that they can be downloaded from GitHub Actions workflow web view as well.
Example of GCS output for `build_source -> upload_source_to_gcs` jobs:
```
gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip
```
Wheel files (`*.whl`) are processed accordingly by `build_wheels -> upload_wheels_to_gcs` jobs.
Example of GCS output at the end of the workflow:
```
gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mik-laj commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-651359183
🎉
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435403856
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
+ CIBW_BUILD_VERBOSITY: 3
Review comment:
Good point, we don't need it I think. I will remove this line so the default verbosity will be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed edited a comment on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-641886596
@aaltay thank you for review. Answers for your questions:
> on pull_request:
> Can we trigger this with a phrase? And/Or limit it to python changes?
I added `paths` for filtering so only meaningful changes will trigger build check.
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths.
> What is gh-action artifacts? How does one access this to get the artifacts?
GitHub Action artifacts allow to persist data on storage space to share that data with another job.
They can be downloaded from GitHub UI for by API. GitHub stores artifacts for 90 days
You can test it on example run: https://github.com/TobKed/beam/actions/runs/129158855
[GitHub Actions Artifacts documentation](https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts#about-workflow-artifacts)
[GitHub API - Download an artifact](https://developer.github.com/v3/actions/artifacts/#download-an-artifact)
> How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)
It is done automatically in `.gihub/workflows/cancel.yml` with usage of https://github.com/styfle/cancel-workflow-action.
This action is checking is there any other previously triggered and running workflow on given branch and cancel them.
It saves resources and prevents colliding actions on GCS bucket.
> What is release-candidate branches? It is release-* ? (If yes, this looks good.)
I think I made mistake here. To be precise `release-*` branch is called `RELEASE_BRANCH` (based on [build_release_candidate.sh#L57](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L57)) and based on [build_release_candidate.sh#L110](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L110) proper pattern for release-candidate branches should be: `v*` (previously this pattern was for `tags` only)
Edit: `v*` seems to be too general for branch, i change it to `v**-RC*`
> Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?
Currently source files are build here [build_release_candidate.sh#L174](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L174) based on RELEASE_BRANCH, the release branch is created few lines above [build_release_candidate.sh#L109](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L109), this push will trigger build of python sources and wheels.
Build can be triggered in both ways: by pushing to branches matching patterns or creating PRs to these branches (however due to path filtering build on pull request may be not triggered).
I hope it answers your questions :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-648097757
> @TobKed - please ping me if this is ready to be merged.
@aaltay I think it is ready, however I have some questions before it can me merged:
1. Who can can and should setup secrets for GitHub Repository? I described necessary secret variables in PR description (**Before merging**) ([docs](https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets))
2. Who can can and should setup versioning and lifecycle management for GCS Bucket?
3. What do you think about not storing `GCP_BUCKET` as the secret but hardcoding it, as it is in `beam-wheels` repository?
Staging bucket is publicly available and not using secret will allow to show full path to the files in the github actions logs.
It will make easier to update/modify sh scripts for release process as well.
I propose to use `beam-wheels-staging` bucket as it is in `beam-wheels`.
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-651260252
Looks everything is fine 😀
https://github.com/apache/beam/actions/runs/151717014
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-641482606
This looks nice. I have a few clarifiying questions:
on pull_request: This is good. Would it trigger on every pull request. This may not be needed. I am not sure what GH resource we will have and what kind of queue there will be. We would not want ot add test load to all PRs.
Questions:
- Can we trigger this with a phrase? And/Or limit it to python changes?
- What is gh-action artifacts? How does one access this to get the artifacts?
on push:
Question:
- What is release-candidate branches? It is release-* ? (If yes, this looks good.)
on schedule: this looks good.
Other questions:
- How do we use cancel previous runs workflow? Does it work automatically? (If yes, this is great.)
> Nightly is addressed by on schedule trigger.
+1.
> I investigated triggering builds manually by using repository_dispatch and curl however build triggered by it is executed on master branch and I think there is no convenient way to run it on different branches. If your question is related to "Build Release Candidate" phase it may be solved here as well since build will be triggered during pushing to master/release/release-candidate branches (and related to it by commit hash).
Yes, I think this works. My question was how could a release manager build wheels. Release manager will push some commits but they will not do this using a PR. Would it still trigger builds? And release manager might want to build release at a specific commit in the release branch could they do this by opening a PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on a change in pull request #11877: Build python wheels on GItHub actions for Linux and MacOS
Posted by GitBox <gi...@apache.org>.
epicfaace commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r433461511
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,91 @@
+on: [push]
+name: Build python wheels
+
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v1
Review comment:
Should we pin the version of python here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435203092
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
It is uploading sdist output (`zip`,`tag.gz`) files as artifacts where they are picked up by `upload_source_to_gcs` jobs. An additional advantage of the artifacts is fact that they can be downloaded from GitHub Actions workflow web view as well.
Example of GCS output for `build_source -> upload_source_to_gcs` jobs:
```
gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip
```
[here](https://github.com/TobKed/beam/actions/runs/123711355) you can check GitHub Action which ran on my fork.
Wheel files (`*.whl`) are processed accordingly by `build_wheels -> upload_wheels_to_gcs` jobs.
Example of GCS output at the end of the workflow:
```
gs://**/apache-beam-2.23.0.dev0.tar.gz
gs://**/apache-beam-2.23.0.dev0.zip
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp27-cp27mu-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp35-cp35m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp36-cp36m-manylinux2010_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-macosx_10_9_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux1_x86_64.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_i686.whl
gs://**/apache_beam-2.23.0.dev0-cp37-cp37m-manylinux2010_x86_64.whl
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435095271
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
Review comment:
typo
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed edited a comment on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-639667163
cc @potiuk, @brucearctor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-646834753
@TobKed - please ping me if this is ready to be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay merged pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay merged pull request #11877:
URL: https://github.com/apache/beam/pull/11877
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-651251549
Thanks! Sure, I will monitor is everything fine :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-646144887
@aaltay in relation to triggering patters, I think I misunderstood naming conventions. I will make update. Thank you for noticing it.
> This action will create the source package and wheels right? So, it would not rely on the sh files to build a tar ball and push it?
Yes, it will create both. Whole process of building from scratch can be done in the GItHub Actions workflow. I will create another PR dependent on it, which will use GitHub Actions Artifacts and/or GCS in release sh files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436068410
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
Review comment:
Thanks!. Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435212390
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
Review comment:
Since some unix specific c libraries are not present on windows (`unistd.h`, `get_clocktime`) I will work on it in separate branch/PR. I think there will be no need to create seperate file for windows
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-646396874
Thank you @TobKed. I replied to one of your earlier comments. Besides that I think this looks good.
Feel free to ping me if I do not respond to your questions within a few days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435099607
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
+ CIBW_BUILD_VERBOSITY: 3
Review comment:
Do we need higher verbosity normally?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436068604
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: sdks/python/dist
+
+ prepare_gcs:
+ name: Prepare GCS
+ needs: build_source
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Remove existing files on GCS bucket
+ run: gsutil rm -r "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/" || true
+
+ upload_source_to_gcs:
+ name: Upload source to GCS bucket
+ needs: prepare_gcs
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Download wheels
+ uses: actions/download-artifact@v2
+ with:
+ name: source_gztar_zip
+ path: source/
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ service_account_email: ${{ secrets.CCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.CCP_SA_KEY }}
+ - name: Copy sources to GCS bucket
+ run: gsutil cp -r -a public-read source/* gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/
+ - name: List sources on GCS bucket
+ run: |
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.tar.gz"
+ gsutil ls "gs://${{ secrets.CCP_BUCKET }}/${GITHUB_REF##*/}/*.zip"
+
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}
+ needs: prepare_gcs
+ runs-on: ${{ matrix.os }}
+ strategy:
+ matrix:
+ os : [ubuntu-18.04, macos-10.15]
+ steps:
+ - name: Download source
+ uses: actions/download-artifact@v2
+ with:
+ name: source
+ path: apache-beam-source
+ - name: Install Python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Install packages on Mac
+ if: startsWith(matrix.os, 'macos')
+ run: |
+ brew update
+ brew install pkg-config
+ - name: Install cibuildwheel
+ run: pip install cibuildwheel==1.4.2
+ - name: Build wheel
+ working-directory: apache-beam-source
+ env:
+ CIBW_BUILD: cp27-* cp35-* cp36-* cp37-*
+ CIBW_BUILD_VERBOSITY: 3
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-649311302
HI @aaltay and @tysonjh
Big thanks for helping me with this.
According to JIRA secrets are set 🎉
https://issues.apache.org/jira/browse/INFRA-20459
Please take a final look on the code before merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r435107715
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
Review comment:
I think we don't need `cython` at this point, since it's installed and used in `build_wheels` job. Most probably the same applies to `wheels`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11877:
URL: https://github.com/apache/beam/pull/11877#issuecomment-648280381
> > @TobKed - please ping me if this is ready to be merged.
>
> @aaltay I think it is ready, however I have some questions before it can me merged:
>
> 1. Who can can and should setup secrets for GitHub Repository? I described necessary secret variables in PR description (**Before merging**) ([docs](https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets))
I do not have access to the "Settings" tab. Could you work with @tysonjh and infra on this one?
> 2. Who can can and should setup versioning and lifecycle management for GCS Bucket?
I can give you access to apache-beam-testing project to update things. Or I can do it. Let me know.
> 3. What do you think about not storing `GCP_BUCKET` as the secret but hardcoding it, as it is in `beam-wheels` repository?
> Staging bucket is publicly available and not using secret will allow to show full path to the files in the github actions logs.
> It will make easier to update/modify sh scripts for release process as well.
> I propose to use `beam-wheels-staging` bucket as it is in `beam-wheels`.
Sounds good to me.
>
> WDYT?
@tysonjh could help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #11877: [BEAM-10184] Build python wheels on GitHub Actions for Linux/MacOS
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r443506463
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
>I think it would be much cleaner and explicit. Did I understand correctly your idea?
Yes. Your understanding is correct. This sounds good to me.
I renamed steps. Please take a look is there no mistakes.
> Clean up:
Both options are good. Can we do both?
Sure. We can do both.
Related to versioning and lifecycle management: I added information in the PR description.
For periodic cleaning tasks I created separate dependent draft PR: https://github.com/apache/beam/pull/12049
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org