You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vincbeck (via GitHub)" <gi...@apache.org> on 2024/02/23 21:11:02 UTC
[PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
vincbeck opened a new pull request, #37666:
URL: https://github.com/apache/airflow/pull/37666
Implement `filter_permitted_dag_ids` in AWS auth manager. This method leverages `batch_is_authorized` API from Amazon Verified Permissions to check DAG access from the user in one call instead of N calls where N is the number of DAGs.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1513063282
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
+ if method in methods:
+ request: IsAuthorizedRequest = {
+ "method": cast(ResourceMethod, method),
+ "entity_type": AvpEntities.DAG,
+ "entity_id": dag_id,
+ }
+ if dag_id not in requests:
+ requests[dag_id] = {}
Review Comment:
Nice! Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1513057239
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
+ if method in methods:
Review Comment:
You cannot iterate over `Container` (I wish I could)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1511932893
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
Review Comment:
Adding on to Niko's comment above, if you use type ResourceMethod instead of a string, this simplifies the rest of this code as well. You only ever use `method` after casting it (twice) anyway. Seems like you might as well use it directly as a ResourceMethod
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1513059689
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
Review Comment:
ResourceMethod is a literal defined here: https://github.com/apache/airflow/blob/main/airflow/auth/managers/base_auth_manager.py#L60
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
Review Comment:
`ResourceMethod` is a literal defined here: https://github.com/apache/airflow/blob/main/airflow/auth/managers/base_auth_manager.py#L60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1511832779
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
+ if method in methods:
Review Comment:
Why not just iterate over `methods`?
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
Review Comment:
Aren't methods supposed to be of type `Container[ResourceMethod]`? Here you're just using strings
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
+ if method in methods:
+ request: IsAuthorizedRequest = {
+ "method": cast(ResourceMethod, method),
+ "entity_type": AvpEntities.DAG,
+ "entity_id": dag_id,
+ }
+ if dag_id not in requests:
+ requests[dag_id] = {}
Review Comment:
Use a DefaultDict?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1513089052
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
Review Comment:
`ResourceMethod` is a Literal, is there another way to initialize a Literal? If I can avoid casting it later, I take it :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1511924543
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
Review Comment:
Yeah. In L370 in base_auth_manager (above) they are using string literals as well though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1518053030
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
Review Comment:
Maybe not. Is the casting just to appease MyPy? It doesn't seem to have any practical effect. I had expected it to at least verify that the value being cast was a valid value...
You can't iterate over a Literal so `for method in Literal["GET", "PUT"]` is out....
Yeah, I guess I retract this comment, sorry about that. Using a Literal there limited our options, I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #37666:
URL: https://github.com/apache/airflow/pull/37666#discussion_r1518176713
##########
airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py:
##########
@@ -443,6 +443,62 @@ def batch_is_authorized_variable(
]
return self.avp_facade.batch_is_authorized(requests=facade_requests, user=self.get_user())
+ def filter_permitted_dag_ids(
+ self,
+ *,
+ dag_ids: set[str],
+ methods: Container[ResourceMethod] | None = None,
+ user=None,
+ ):
+ """
+ Filter readable or writable DAGs for user.
+
+ :param dag_ids: the list of DAG ids
+ :param methods: whether filter readable or writable
+ :param user: the current user
+ """
+ if not methods:
+ methods = ["PUT", "GET"]
+
+ if not user:
+ user = self.get_user()
+
+ requests: dict[str, dict[ResourceMethod, IsAuthorizedRequest]] = {}
+ requests_list: list[IsAuthorizedRequest] = []
+ for dag_id in dag_ids:
+ for method in ["GET", "PUT"]:
Review Comment:
No worries! hat was legitimate question. Yeah the `cast` is to basically make MyPy happy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #37666:
URL: https://github.com/apache/airflow/pull/37666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org