You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vincbeck (via GitHub)" <gi...@apache.org> on 2023/02/16 19:42:33 UTC
[GitHub] [airflow] vincbeck opened a new pull request, #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
vincbeck opened a new pull request, #29580:
URL: https://github.com/apache/airflow/pull/29580
Some AWS customers are complaining about their Secrets Manager and/or Systems manager bills because each connection/variable/config send request to Secrets Manager and/or Systems manager (if `*_prefix` is not null). The backend is accessed for each variable and connection access and it is done every time the Scheduler parses DAGs, resulting in billions of request for some customers every month. This also results in extra burden on scheduler as Secrets manager/Systems Manager is accessed each time the DAG is parsed.
Adding `*_lookup_pattern` parameters to allow the user to specify which connection/variable/config he wants to be looked in Secrets Manager and/or Systems manager.
Related: #19251
@Taragolis you might be interested to look into this one.
AWS folks: @ferruzzi, @vandonr-amz, @syedahsn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433693955
I'm sorry but I don't follow which problem are we tackling here.
The PR description talks about DAG parsing but @shubham22 comment is not about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1450717211
@eladkal marking for 2.6.0 as this sounds more like an improvement/feature to me. Feel free to update if I missed something :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1443984840
> Should we modify a bit the description of https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/secrets-backend/index.html#search-path ?
>
> > If you enable an alternative secrets backend, it will be searched first, followed by environment variables, then metastore. **This search ordering is not configurable**.
>
> When merging this PR the statement becomes not very accurate
Good point. I added a sentence saying that, depending on the secret backend used, there might be an option to filter which connection/variable/config is searched in the secret backend. Let me know if it does look good to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1450763713
Thanks @eladkal for the details, I completely missed that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433630542
> for each variable and connection access and it is done every time the Scheduler parses DAGs
It happen every time if users not follow Best Practices about [Top Level Python Code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code) 🤣 . Or some of the Operators call establish connections in constructor, I know only about SSHOperator from Community Providers with this behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #29580:
URL: https://github.com/apache/airflow/pull/29580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1450426655
@dimberman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433760620
Sorry for the bad description guys! I definitely did not expect so much confusion so sorry for that again. I updated the description which, hopefully, make the PR more clear.
I understand we should push people to write DAGs which follow guidelines but I am definitely not targeting specifically these users when doing this change. See the example in the description. And I feel there are a lot of other use cases which this feature can be a good add-on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1450733661
@pierrejeambrun this needs to go to 2.5.2
from core point of view this PR contains only doc so this is a doc-change only.
The core logic is of providers release cycle which will contain the change as feature release for of the Amazon provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433683904
Maybe it's time resurrect this one from @Taragolis ? https://github.com/apache/airflow/pull/23560
I have a feeling that adding lookup filter is quite a bit more complex (in terms of users understanding what to do there properly and in terms of usage protection). What happens if you change the pattern ? Will you remember to update it or you will be surprised by high involce next month?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433725441
> Customers don't always write the most optimized DAGs. Conns, Configs and variables are always searched in secrets backend before env variables and meta store. This change will allow customers to only lookup secrets backend when their variables, et al. follow a pattern defined by them. Based on MWAA data, I can confirm that customers are spending hundreds of dollars on their secrets backed, just to use it with Ariflow.
That actually just postpone the problem, and might lead to problem with Aurora instances (AFAIK it uses in MWAA as Postgres backend), yeah it would be cheaper for end users, because in this case they would not pay for IOPS on Aurora in case of MWAA, but still can be an issue with backend itself.
Is MWAA still support only Secrets Manager by default? Because I guess SSM Parameter store much cheaper rather than AWS SM. You don't have to pay 0.4 cents secrets/per month, and no cost for API call for Standard Parameter with Standard Throughput, I know because personally use SSM since Secrets Backends introduced in Airflow 1.10.10
I do not have any concern about this changes, just want to mention that the main problem, that users do not follow Best Practice, and as result they have a huge or not bills. The huge bill might happen if user turn on DEBUG level in production and write everything to logs in this case CloudWatch pricing might be, well... pretty surprising.
> Maybe it's time resurrect this one from @Taragolis ? https://github.com/apache/airflow/pull/23560
Ooooh... My first PR to Airflow repo 🤣 That still in my list, actually in two list, TODO and make wide discussion about Configurations and consistency with Secrets Backend, still no idea how to resolve the problem, that Airflow DB use both Secrets Backend capability and own implementation for obtain Variables and Connections in CLI/API/Webserver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1481733865
Marked for 2.6, conflicting and requires 2 providers only to merge the 1 line core doc change. Raise a specific PR to v2-5 branch to avoid cherry picking them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1450738834
@dimberman I'm going to merge this PR so we can start working on providers + core release. If something else is needed we can address it in followup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
shubham22 commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433667203
Customers don't always write the most optimized DAGs. Conns, Configs and variables are always searched in secrets backend before env variables and meta store. This change will allow customers to only lookup secrets backend when their variables, et al. follow a pattern defined by them. Based on MWAA data, I can confirm that customers are spending hundreds of dollars on their secrets backed, just to use it with MWAA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
shubham22 commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433698397
> The PR description talks about DAG parsing but @shubham22 comment is not about that.
Sorry for the confusion, Elad. @vincbeck will soon update the description to make the scope and impact clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1442507073
Should we modify a bit the description of https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/secrets-backend/index.html#search-path ?
> If you enable an alternative secrets backend, it will be searched first, followed by environment variables, then metastore. **This search ordering is not configurable**.
When merging this PR the statement becomes not very accurate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1439024161
Thanks @dimberman for the feedbacks. I addressed them, feel free to review when you get a chance :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1437592624
Yep. I thouthg about it and this seems like non-obvious great way of solving the "cost" problem connected with secret backends.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on a diff in pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "dimberman (via GitHub)" <gi...@apache.org>.
dimberman commented on code in PR #29580:
URL: https://github.com/apache/airflow/pull/29580#discussion_r1111314542
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -84,12 +88,24 @@ class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin):
:param connections_prefix: Specifies the prefix of the secret to read to get Connections.
If set to None (null value in the configuration), requests for connections will not be
sent to AWS Secrets Manager. If you don't want a connections_prefix, set it as an empty string
+ :param connections_lookup_pattern: Specifies a pattern the connection ID needs to match to be looked up in
+ AWS Secrets Manager. Applies only if `connections_prefix` is not None.
+ If set to None (null value in the configuration), all connections will be looked up first in
+ AWS Secrets Manager.
Review Comment:
I really like this idea. Cool feature add on!
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -264,14 +286,14 @@ def get_conn_uri(self, conn_id: str) -> str | None:
def get_variable(self, key: str) -> str | None:
"""
- Get Airflow Variable from Environment Variable
+ Get Airflow Variable
Review Comment:
Can you add some more description here. If we're getting this variable from multiple places maybe we should be clear about the options?
##########
airflow/providers/amazon/aws/secrets/systems_manager.py:
##########
@@ -166,16 +185,26 @@ def get_config(self, key: str) -> str | None:
if self.config_prefix is None:
return None
- return self._get_secret(self.config_prefix, key)
+ return self._get_secret(self.config_prefix, key, self.config_lookup_pattern)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _get_secret(self, path_prefix: str, secret_id: str, lookup_pattern: str | None) -> str | None:
"""
Get secret value from Parameter Store.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
+ :param lookup_pattern: If provided, `secret_id` must match this pattern to look up the secret in
+ Systems Manager
"""
+ if lookup_pattern is not None and not re.match(lookup_pattern, secret_id, re.IGNORECASE):
Review Comment:
```suggestion
if lookup_pattern and not re.match(lookup_pattern, secret_id, re.IGNORECASE):
```
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -282,14 +304,19 @@ def get_config(self, key: str) -> str | None:
if self.config_prefix is None:
return None
- return self._get_secret(self.config_prefix, key)
+ return self._get_secret(self.config_prefix, key, self.config_lookup_pattern)
- def _get_secret(self, path_prefix, secret_id: str) -> str | None:
+ def _get_secret(self, path_prefix, secret_id: str, lookup_pattern: str | None) -> str | None:
"""
Get secret value from Secrets Manager
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
+ :param lookup_pattern: If provided, `secret_id` must match this pattern to look up the secret in
+ Secrets Manager
"""
+ if lookup_pattern is not None and not re.match(lookup_pattern, secret_id, re.IGNORECASE):
Review Comment:
```suggestion
if lookup_pattern and not re.match(lookup_pattern, secret_id, re.IGNORECASE):
```
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -41,13 +42,16 @@ class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin):
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections"}
- For example, if secrets prefix is ``airflow/connections/smtp_default``, this would be accessible
- if you provide ``{"connections_prefix": "airflow/connections"}`` and request conn_id ``smtp_default``.
- If variables prefix is ``airflow/variables/hello``, this would be accessible
- if you provide ``{"variables_prefix": "airflow/variables"}`` and request variable key ``hello``.
- And if config_prefix is ``airflow/config/sql_alchemy_conn``, this would be accessible
- if you provide ``{"config_prefix": "airflow/config"}`` and request config
- key ``sql_alchemy_conn``.
+ For example, when ``{"connections_prefix": "airflow/connections"}`` is set, if a secret is defined with
+ the path ``airflow/connections/smtp_default``, the connection with conn_id ``smtp_default`` would be
+ accessible.
+
+ When ``{"variables_prefix": "airflow/variables"}`` is set, if a secret is defined with
+ the path ``airflow/variables/hello``, the variable with they ``hello`` would be accessible.
Review Comment:
```suggestion
the path ``airflow/variables/hello``, the variable with the name``hello`` would be accessible.
```
##########
airflow/providers/amazon/aws/secrets/systems_manager.py:
##########
@@ -166,16 +185,26 @@ def get_config(self, key: str) -> str | None:
if self.config_prefix is None:
return None
- return self._get_secret(self.config_prefix, key)
+ return self._get_secret(self.config_prefix, key, self.config_lookup_pattern)
- def _get_secret(self, path_prefix: str, secret_id: str) -> str | None:
+ def _get_secret(self, path_prefix: str, secret_id: str, lookup_pattern: str | None) -> str | None:
"""
Get secret value from Parameter Store.
:param path_prefix: Prefix for the Path to get Secret
:param secret_id: Secret Key
+ :param lookup_pattern: If provided, `secret_id` must match this pattern to look up the secret in
+ Systems Manager
"""
+ if lookup_pattern is not None and not re.match(lookup_pattern, secret_id, re.IGNORECASE):
+ return None
+
ssm_path = self.build_path(path_prefix, secret_id)
+
+ # AWS Systems Manager mandate to have a leading "/". Adding it dynamically if not there
+ if not ssm_path.startswith("/"):
+ ssm_path = f"/{ssm_path}"
Review Comment:
Maybe break this out into a function? I know it's a small nit, but would be a bit cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #29580:
URL: https://github.com/apache/airflow/pull/29580#discussion_r1113446539
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -264,14 +286,14 @@ def get_conn_uri(self, conn_id: str) -> str | None:
def get_variable(self, key: str) -> str | None:
"""
- Get Airflow Variable from Environment Variable
+ Get Airflow Variable
Review Comment:
In the context of this class, we are actually getting it from only one location: AWS Secrets Manager. If this function returns None, then it is fetched from other location: Environment variable then metastore but this is done outside of this class. [See documentation here](https://airflow.apache.org/docs/apache-airflow/1.10.10/howto/use-alternative-secrets-backend.html). Let me know if you still think I should update the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433652413
I agree with @Taragolis the problem as described in this PR is not the same as described in https://github.com/apache/airflow/issues/19251 in the issue dag parsing is not in the problem domain.
@vincbeck Please share DAG code example that this PR attempts to solve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shubham22 commented on pull request #29580: Allow to specify which connection, variable or config are being looked up in the backend using *_lookup_pattern parameters
Posted by "shubham22 (via GitHub)" <gi...@apache.org>.
shubham22 commented on PR #29580:
URL: https://github.com/apache/airflow/pull/29580#issuecomment-1433690605
> in terms of users understanding what to do there properly and in terms of usage protection
This is a fair argument and agree that this can be misused or can lead to issues which are hard to track. We did consider adding only prefix, say "aws-conn-* instead of general regex, but Airflow customers are primed toward flexibility. This is why we went with regex to make it suitable for different needs. With documentation and, likely, a blogpost we will try to share with customers best practices around this. Like many other things, if customers really want to break, there are many ways to break it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org