You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/06 12:36:10 UTC
[GitHub] [airflow] ashb opened a new pull request #13514: WIP: Test out github actions via gitmodules
ashb opened a new pull request #13514:
URL: https://github.com/apache/airflow/pull/13514
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756073279
Yeah -for _adding_ a new submodule it shows that. Updating a submodule shows the full/better view https://github.com/ashb/airflow/pull/3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-757468365
Will do first thing Monday
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RA80533 commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
RA80533 commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855226302
@ashb What prompted this from being introduced? This is the only project I've come across that partitions their third-party Actions in such a manner. Was there any prior attempts at this anywhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756076802
Unrelated to this PR, but I thought that manually setting "full tests needed" label on my PR https://github.com/ashb/airflow/pull/3 and then re-running it should have made it run everything, but it still only ran minimal tests
Fixed by https://github.com/apache/airflow/pull/13538
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13514: WIP: Test out github actions via gitmodules
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#discussion_r552625251
##########
File path: .gitmodules
##########
@@ -0,0 +1,3 @@
+[submodule ".github/actions/get-workflow-origin"]
+ path = .github/actions/get-workflow-origin
+ url = https://github.com/potiuk/get-workflow-origin
Review comment:
BTW. This change should use SHA not master version otherwise we (implicitly) fall in the same trap as https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
This is why subrepo is also much nicer because you always have specific commit of the repo you are linking to. You cannot link to branch - you bring very specific version of the code in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756072638
This is what I see rather than list of files:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13514: WIP: Test out github actions via gitmodules
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#discussion_r552658954
##########
File path: .gitmodules
##########
@@ -0,0 +1,3 @@
+[submodule ".github/actions/get-workflow-origin"]
+ path = .github/actions/get-workflow-origin
+ url = https://github.com/potiuk/get-workflow-origin
Review comment:
I see- so it is stored in the git tree https://stackoverflow.com/questions/5033441/where-does-git-store-the-sha1-of-the-commit-for-a-submodule . That's fine. So then it's great that GitHub shows the actual files in review then. otherwise it woudl have been even more difficult to review :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756072638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: WIP: Test out github actions via gitmodules
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-755278359
See https://github.com/astronomer/airflow/pull/1171/files for what the review process looks like for when a submodule is updated in a PR. (I downgraded that submodule to v1_2 in that draft PR)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756100108
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756074437
> Are you sure it works from the forks? I suspect it is only for PRs in the same repo.
I will push it as a branch to 'apache/airflow' and see if it looks better. We can have a rule that only committers make changes to actions and only do it via internal PR if we find that this is the case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #13514: WIP: Test out github actions via gitmodules
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#discussion_r552633851
##########
File path: .gitmodules
##########
@@ -0,0 +1,3 @@
+[submodule ".github/actions/get-workflow-origin"]
+ path = .github/actions/get-workflow-origin
+ url = https://github.com/potiuk/get-workflow-origin
Review comment:
It doesn't -- that is now how submodules work. It has a pinned sha, just not in this file -- it's a "property" of the submodule path itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756071711
I do not see the code from `submodules` in the review @ashb . Do you know why it is so ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756072638
This is what I see rather than list of files:
Not a deal breaker. But would be nice to have it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855227987
@RA80533 this brought on a setting by Apache Software Foundation Infra and security teams changed on our GitHub org - they only allow Actions from official GitHub, or ASF repos.
This is to somewhat defend against "supply chain" attacks where a project uses some random action which is then updated to exfil creds etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756037859
Could someone please double check my changes are right, in particular the build-images one -- I think it should be running the actions from the main branch, not the PR branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756069412
Ah cool yes, I'll check that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855227731
> @ashb How did this idea come about? This is the only project I've come across that partitions their third-party Actions in such a manner. Was there any prior attempts at this anywhere else?
This was triggered by a potential supply-chain attack that can be issued when you are integrating GitHub Actions via tag or branch. Submodule solution automatically implements all the recommendations described by Github Actions here https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions , including the fact that it is very easy to review any PRs where actions are updated (such PR will allow to review the changes in the action as submodules are nicely integrated in GitHub PR review process).
You can read summary of the GitHub Action status here - and all context and explanations for recommendations we came up with and proposed to other ASF projects:
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756070598
Running here: https://github.com/potiuk/airflow/actions/runs/468787325
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-757458488
Hey @ashb - can you rebase please :) ? I think the errors have been solved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855228055
About prior art - not that I am aware of it but also Apache beam started using it after the discussion https://github.com/apache/beam/blob/master/.gitmodules
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756069026
> Could someone please double check my changes are right, in particular the build-images one -- I think it should be running the actions from the main branch, not the PR branch.
You can check it yourself, just push to your fork as main branch (you have to have github actions enabled). I will do it now as well, but this is the easiest one for you to check it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756071986
Hmm, what do I have to do to turn actions on for my fork? I don't see an option to enable/disable actions in my settings. Must be missing it.
Oh found it - the workflow was manually disabled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756071986
Hmm, what do I have to do to turn actions on for my fork? I don't see an option to enable/disable actions in my settings. Must be missing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756076802
Unrelated to this PR, but I thought that manually setting "full tests needed" label on my PR https://github.com/ashb/airflow/pull/3 and then re-running it should have made it run everything, but it still only ran minimal tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#discussion_r553288934
##########
File path: .github/workflows/build-images-workflow-run.yml
##########
@@ -297,8 +302,19 @@ jobs:
else
echo "::set-output name=proceed::true"
fi
+ - name: >
Review comment:
That is indeed important to use main-airflow here for security reasons!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #13514:
URL: https://github.com/apache/airflow/pull/13514
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855227731
> @ashb How did this idea come about? This is the only project I've come across that partitions their third-party Actions in such a manner. Was there any prior attempts at this anywhere else?
This was triggered by a potential supply-chain attack that can be issued when you are integrating GitHub Actions via tag or branch. Submodule solution automatically implements all the recommendations described by Github Actions here https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions , including the fact that it is very easy to review any PRs where actions are updated (such PR will allow to review the changes in the action as submodules are nicely integrated in GitHub PR review process).
You can read summary of the GitHub Action status here - and all context and explanations for recommendations we came up with and proposed to other ASF projects:
https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756076142
> Cross repo PR that updates a sub-module ^^
Yep. just tested it :). All Good !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756074555
@potiuk https://github.com/astronomer/airflow/pull/1174
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756074555
@potiuk https://github.com/astronomer/airflow/pull/1174
Cross repo PR that updates a sub-module ^^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-756073688
> Yeah -for _adding_ a new submodule it shows that. Updating a submodule shows the full/better view [ashb#3](https://github.com/ashb/airflow/pull/3)
Are you sure it works from the forks? I suspect it is only for PRs in the same repo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RA80533 edited a comment on pull request #13514: Use git submodules for (securely) using third party Github Actions.
Posted by GitBox <gi...@apache.org>.
RA80533 edited a comment on pull request #13514:
URL: https://github.com/apache/airflow/pull/13514#issuecomment-855226302
@ashb How did this idea come about? This is the only project I've come across that partitions their third-party Actions in such a manner. Was there any prior attempts at this anywhere else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org