You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/01/28 13:17:00 UTC

[PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

potiuk opened a new pull request, #37057:
URL: https://github.com/apache/airflow/pull/37057

   In our process, we generally do not let the scripts in the "build images" workflow to use `scripts/ci`, `dev` and `action` scripts to come from the PR. This is a security feature that prevent Pull Requests from forks to run code on a worker that can potentially access sensitive information - such as GITHUB_TOKEN with write access to Github Registry.
   
   This, however, causes troubles, because in order to test any changes in those scripts affecting building image, you have to close your PR from the fork and push one directly to Apache repository (there in-line build workflows are used from "Test" workflow and those PRs are safe to run, because only committers can push directly to the `apache/airflow` repository branches.
   
   This PR changes default behaviour for committer PRs. Rather than do the same as "regular" PRs, those PRs will not use scripts from the target branch, instead they will use scripts from the incoming PRs of the committers. This is equally safe as running PRs from the `apache/airflow` branch - because we have a reviewed list of committers in our code and "selective checks" job that checks it is run always in the context and with the code of the "target" branch, which means that you cannot manipulate the list of actors.
   
   The Girhub actor is retrieved from pull requests github context (event/pull_request/user/login) so it is impossible to spoof it by the incoming PR.
   
   As part of this PR - list of available selective checks and documentation of PR labels and selective checks (wrongly named as "static checks") were reviewed and updated.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468921710


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   And yes. The answer is `yes` :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37057:
URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913692464

   All right I did some testing and applied a few smal fixes (optimiized away the checkout for target-branch for commiter build as it turned out to be not needed
   
   For anyone looking here (and I would love at least 2 reviews) for security reasons.
   
   I've run some simulated builds from my higrys -> potiuk PRs after pushing the branch to main:
   
   ## Here is an example of committer-originated build:
   
   * Selective check https://github.com/potiuk/airflow/actions/runs/7687733181/job/20948113127?pr=287#step:9:88
   * Image build (skips the script/dev/action replace) https://github.com/potiuk/airflow/actions/runs/7687733181/job/20948122563?pr=287
   
   ## Example where non-committer is the actor in PR
   
   * Selective check: https://github.com/potiuk/airflow/actions/runs/7687718985/job/20948083436?pr=289#step:9:87
   * Image build (scripts/dev/actions) are replaced with the target ones: https://github.com/potiuk/airflow/actions/runs/7687718985/job/20948089594?pr=289#step:5:1
   
   
   The builds here (you can see different actors):
   
   
   <img width="577" alt="Screenshot 2024-01-28 at 20 03 26" src="https://github.com/apache/airflow/assets/595491/bb6ff88c-9a13-4437-b6a3-ecfd6cc15844">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37057:
URL: https://github.com/apache/airflow/pull/37057


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468931589


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   Great!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468866371


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   No there is no way, but.... aI also realized that this change is wrong because the "selective checks" are run AFTER teh scripts are restored... Let me change it and explain :)
   
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37057:
URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913626242

   I will do some more testing in my own fork, to see if I have everything righ. 
   
   And I'd appreciate a bit more thorough review for that one. This is a sensitive part of our actions and it would be good to scrutinize it a bit more :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468868060


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   Ah yeah... That's quite a bit more complex than that ... I will convert that one to draft (this was not a security risk, but it would not work either).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #37057:
URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913711280

   Well done!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37057:
URL: https://github.com/apache/airflow/pull/37057#issuecomment-1914380272

   cc: @ephraimbuddy FYI -> this is something I wil cherry-pick shortly (together with other recent refactors) to v2-8-test and do some testing if it works as expected (there are some smalll caveats in retrieving DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH that I need to test there, so don't be surprised :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468865093


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   Is there any protection to prevent the no-committer user from changing this value to True?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37057:
URL: https://github.com/apache/airflow/pull/37057#discussion_r1468880404


##########
dev/breeze/src/airflow_breeze/utils/selective_checks.py:
##########
@@ -1039,3 +1040,9 @@ def providers_compatibility_checks(self) -> str:
                 if check["python-version"] in self.python_versions
             ]
         )
+
+    @cached_property
+    def is_committer_build(self):
+        if NON_COMMITTER_BUILD_LABEL in self._pr_labels:
+            return False
+        return self._github_actor in COMMITTERS

Review Comment:
   OK. I applied a number of cleanups and clairifcations and comments in the workflow describing what's going on. I also found out along the way that the part where DEFAULT_BRANCH and DEFAULT_CONSTRAINTS_BRANCH retrieval can be simplified, after some past changes in the way how "pull request workflow" uses now target_branhc by default and not "main" as it used to be, so we do not have to use evn variables to pass it now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Allow committer builds to use scripts/ci, dev and actions from the PR [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37057:
URL: https://github.com/apache/airflow/pull/37057#issuecomment-1913594227

   Once we merge that one, committers will have no more the problem that their PRs modifying `dev` scripts will have to close it and re-open it from the `apache` repository. It should **just work**.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org