You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/17 00:27:59 UTC
[GitHub] [airflow] milton0825 opened a new pull request #8889: Support custom Variable implementation
milton0825 opened a new pull request #8889:
URL: https://github.com/apache/airflow/pull/8889
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-629744199
You can override variable using secret backend:
https://airflow.readthedocs.io/en/latest/howto/use-alternative-secrets-backend.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631483938
> Of course there is more than one project that does the same thing called Vault. Of course.
>
> Even with Azure Vault not being granular enough, this feature is still a -1 from me. I'd be happy to see the ability to CUD connections/secrets via the existing Secrets backends -- this is already our interface to a "custom" backend of Variables. We should not have another.
Yep. Fully agree on that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187
Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write.
It seems like `Variable` is now tightly coupled with `SecretBackend` for DAGs to retrieve secrets from Vault. However, I have seen Airflow users that use `Variable` to share data between tasks/DAGs (yes they can also use xcom but they still can choose to use `Variable`) so it's usage is not limited to retrieving secrets. As long as `Variable` support a set method, it can be confusing to the users that they thought they can set a variable backed by Vault, which they actually can't.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631885153
I think maybe it's the right time to raise that as a separate thread on the devlist so that other people can state their opinion as well?
I agree with @ashb and @kaxil that creating a custom variable implementation in this form might be super confusing and in the form proposed it's -1 from me.
Regarding the case you mentioned - using variables for sharing data between tasks - I believe that's not what it is intended for and it should be strongly discouraged. Variables should be relatively static, changeable externally. Using variables for sharing data between tasks breaks the idempotency mechanism built-in Airflow via Xcom (separate value stored for each dag run).
If you want to use variable and maintain idempotency you have to reproduce the xcom apprach - and make your variable key contain the dag_run_id which I think many people would not do that or even think about if they use a convenient variable mechanism. It would have to create multiple variables with a similar name + dag_run_id. That would result in the number of variables growing continuously over time - I think this is a very, very bad use of variables in this case - and having variables effectively read-only with secrets is actually a nice way to avoid this behavior.
In the example from (a bit related but not the same as mentioned by @ashb ) the discussion about hooks (https://lists.apache.org/thread.html/re2bd54b0682e14fac6c767895311baf411ea10b18685474f7683a2f5%40%3Cdev.airflow.apache.org%3E). I created a similar example where Secret backend could be used for sharing data between tasks. But then it would be custom implementation, and we can encourage using pattern where dag_run is part of the secret key name, or even better - you can generate and keep the name of the secret in the XCOM. This is the "standard" Airflow pattern for sharing data between tasks and once we have separate hooks for all secret backend, it is something that should be used rather than variables if you would like to keep the shared data "secret".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 closed pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 closed pull request #8889:
URL: https://github.com/apache/airflow/pull/8889
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631394474
@potiuk I think the point here is to be able to manage Variables by the existing UI screens, so Hooks is the wrong tool for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-629749913
@mik-laj Thanks for sharing the info. `Variable` via `SecretBackend` can only read values and I am not sure if we should extend `SecretBackend` to support write due to security concerns.
Therefore, I think allowing people to provide custom `Variable` implementation that support read/write should fit in this use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631486133
What's the use-case of writing Variables to an external backend?
The main reason for Secret Bakend was to move this responsibility to a product/service that does that nicely. Most of the companies use a single Secrets Manager whether it is Hashicorp Vault, GCP Secrets Manager etc. And the main reason for doing this is that they could have full control over it and those same secrets could be used across multiple deployments.
Even if we allow the feature to write to any Secrets Manager, I doubt many companies will use it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631885153
I think maybe it's the right time to raise that as a separate thread on the devlist so that other people can state their opinion as well?
I agree with @ashb and @kaxil that creating a custom variable implementation in this form might be super confusing and in the form proposed it's -1 from me.
Regarding the case you mentioned - using variables for sharing data between tasks - I believe that's not what it is intended for and it should be strongly discouraged. Variables should be relatively static, changeable externally. Using variables for sharing data between tasks breaks the idempotency mechanism built-in Airflow via Xcom (separate value stored for each dag run).
If you want to use variable and maintain idempotency you have to reproduce it and make your variable key contain the dag_run_id which I think many people would not do that or even think about if they use a convenient variable mechanism. It would have to create multiple variables with a similar name + dag_run_id. That would result in the number of variables growing continuously over time - I think this is a very, very bad use of variables in this case - and having variables effectively read-only with secrets is actually a nice way to avoid this behavior.
In the example from (a bit related but not the same as mentioned by @ashb ) the discussion about hooks (https://lists.apache.org/thread.html/re2bd54b0682e14fac6c767895311baf411ea10b18685474f7683a2f5%40%3Cdev.airflow.apache.org%3E). I created a similar example where Secret backend could be used for sharing data between tasks. But then it would be custom implementation, and we can encourage using pattern where dag_run is part of the secret key name, or even better - you can generate and keep the name of the secret in the XCOM. This is the "standard" Airflow pattern for sharing data between tasks and once we have separate hooks for all secret backend, it is something that should be used rather than variables if you would like to keep the shared data "secret".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-629749913
@mik-laj Thanks for sharing the info. `Variable` via `SecretBackend` can only read values and I don't think we should extend `SecretBackend` to support write due to security concerns.
Therefore, I think allowing people to provide custom `Variable` implementation that support read/write should fit in this use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631486133
This is exactly the reason for creating Secrets Backend where currently it can retrieve both Variables & Connection.
What's the use-case of writing Variables to an external backend?
The main reason for Secret Bakend was to move this responsibility to a product/service that does that nicely. Most of the companies use a single Secrets Manager whether it is Hashicorp Vault, GCP Secrets Manager etc. And the main reason for doing this is that they could have full control over it and those same secrets could be used across multiple deployments.
Even if we allow the feature to write to any Secrets Manager, I doubt many companies will use it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631335327
> @mik-laj Thanks for sharing the info. `Variable` via `SecretBackend` can only read values and I don't think we should extend `SecretBackend` to support write due to security concerns.
I agree with that.
> Therefore, I think allowing people to provide custom `Variable` implementation that support read/write should fit in this use case.
In general, I like the idea. However, my main concern is that with both variable and secret backends give users two ways of reading variables but only one to write them. I can imagine that:
a) users will ask which one they should use for reading
b) users will mix both approaches and will be confused by writing variable by VariableBackend and reading it by SecretBackend
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631466930
Not that familiar with Vault myself, but https://www.vaultproject.io/docs/concepts/policies looks like you can set polcies on a per-path basis:
```hcl
# This section grants all access on "secret/*". Further restrictions can be
# applied to this broad policy, as shown below.
path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# Even though we allowed secret/*, this line explicitly denies
# secret/super-secret. This takes precedence.
path "secret/super-secret" {
capabilities = ["deny"]
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631473133
It would be great to have an understanding how much we can rely on granularity being available. But I believe (at least currently) readonly/write access for whole vault is the common denominator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187
Other use cases might be that we can replace the variable backend with other non-vault storage like S3 and support both read/write.
It seems like `Variable` is now tightly coupled with `SecretBackend` for DAGs to retrieve secrets from Vault. However, I have seen Airflow users that use `Variable` to share data between tasks/DAGs (yes they can also use xcom but they still can choose to use `Variable` as long as it exists) so it's usage is not limited to retrieving secrets. As long as `Variable` support a set method, it can be confusing to the users that they thought they can set a variable backed by Vault, which they actually can't.
Should we separate the secret management out from the Variable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187
Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write.
It seems like `Variable` is now tightly coupled with `SecretBackend` for DAGs to retrieve secrets from Vault. However, I have seen Airflow users that use `Variable` to share data between tasks/DAGs (yes they can also use xcom but they still can choose to use `Variable` as long as it exists) so it's usage is not limited to retrieving secrets. As long as `Variable` support a set method, it can be confusing to the users that they thought they can set a variable backed by Vault, which they actually can't.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631522248
I can see people wanting to store connections in (Hashicorp) Vault, but still allow Admins/Ops in the cluster to edit Variables, or add new connections; but they just then store it in Vault so they know how it is secured; that secrets are all in a single place etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631565244
> I can see people wanting to store connections in (Hashicorp) Vault, but still allow Admins/Ops in the cluster to edit Variables, or add new connections; but they just then store it in Vault so they know how it is secured; that secrets are all in a single place etc.
Vaults are more often than not automated using Terraform or either Ansible. Anything in Airflow that writes a secret to a Vault or a Secret Manager would be visible somewhere in the UI (Code View if directly writing using DAG, Xcom if that secret is retrieved from any other location and passed on to a task etc that write to Secret Manager).
Anyways, whether or not to have "write"/"put" method for Secret Backend is a separate discussion. But -1 for me too for this PR, no need for a separate backend for Variable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631619067
> What's the use-case of writing Variables to an external backend?
Say we are in a migration process to move DAGs from an old airflow cluster to new airflow cluster. And many DAGs shared the same airflow variable, some of them reads variable and some of them writes the variable. We need a way to double write the variable to both cluster during the migration, hence the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631620756
> > What's the use-case of writing Variables to an external backend?
>
> Say we are in a migration process to move DAGs from an old airflow cluster to new airflow cluster. And many DAGs shared the same airflow variable, some of them reads variable and some of them writes the variable. We need a way to double write the variable to both cluster during the migration, hence the PR to support that.
Why not writing and running it as custom code? it does seem like your specific case and not a common feature that people would be using.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631359757
@milton0825 - see the discussion I started few days ago about Secret Hooks. https://lists.apache.org/thread.html/re2bd54b0682e14fac6c767895311baf411ea10b18685474f7683a2f5%40%3Cdev.airflow.apache.org%3E
I think Secret Backend is good in its support to read connections and variables - because it is there to keep the airflow configuration. What I think you want to achieve can be done via Secret Hooks - so the usual airflow mechanism to talk to external services. You could very easily write custom operators that will be using hooks to read/write secrets to secret backends. And what's best here - you will be able to use different secret backends to keep Airflow configuration and different to store such "secret" values that should be writeable. I think this is a perfect setup from the security point of view - you do not want to give write access to the secret backend that keeps Airflow configuration to all the jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631483171
Of course there is more than one project that does the same thing called Vault. Of course.
Even with Azure Vault not being granular enough, this feature is still a -1 from me. I'd be happy to see the ability to CUD connections/secrets via the existing Secrets backends -- this is already our interface to a "custom" backend of Variables. We should not have another.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187
Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write.
It seems like `Variable` is now tightly coupled with `SecretBackend` for DAGs to retrieve secrets from Vault. However, I have seen Airflow users that use `Variable` to share data between tasks/DAGs (yes they can also use xcom but they still can choose to use `Variable` as long as it exists) so it's usage is not limited to retrieving secrets. As long as `Variable` support a set method, it can be confusing to the users that they thought they can set a variable backed by Vault, which they actually can't.
Should we separate the secret management out from the Variable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
milton0825 edited a comment on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631619067
> What's the use-case of writing Variables to an external backend?
Say we are in a migration process to move DAGs from an old airflow cluster to new airflow cluster. And many DAGs shared the same airflow variable, some of them reads variable and some of them writes the variable. We need a way to double write the variable to both cluster during the migration, hence the PR to support that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631471582
Not in Azure's Vault for example (and I think this it might be one of the most popular in Corporate world):
https://docs.microsoft.com/en-us/azure/key-vault/general/secure-your-key-vault#key-vault-access-policies
Key Vault access policies grant permissions separately to keys, secrets, and certificate. You can grant a user access only to keys and not to secrets. Access permissions for keys, secrets, and certificates are at the vault level. Key Vault access policies don't support granular, object-level permissions like a specific key, secret, or certificate. To set access policies for a key vault, use the Azure portal, the Azure CLI, Azure PowerShell, or the Key Vault Management REST APIs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631438618
@ashb -> Understood. Agree. And I understand the concerns. Ans I agree that if the configuration for Airlfow is going to be managed by the UI (And readonly set fo False), then it's OK to use existing Secret Backend to do writes.
I can easily imagine both cases
a) readonly = True - where all the secrets are managed externally
b) readonly = False - where the Airflow UI is used to do it
I think current vault Implementation (correct me if I am wrong) are "all-or-nothing" so you can't configure read/write separately for different keys/groups of keys so I think this is the biggest granularity you can do (readonly flag for whole installation). But I think it's good enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #8889: Support custom Variable implementation
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #8889:
URL: https://github.com/apache/airflow/pull/8889#issuecomment-631393939
This is a -1 from me -- it is a hugely overlapping feature with the Secrets backend, and makes it harder for users to configure/know which one to use
> I don't think we should extend SecretBackend to support write due to security concerns.
What security concerns? Please be specific.
All of the "real" secrets backends (Vault, AWS, GCP etc) allow fairly tight and granular control of permissions extnerally to Airflow, so if people want to use Vault and not let Airflow manage it then they can deny permissions there.
Additionally we could have a (default) flag to the secrets backends of `readonly=True` so that we don't allow editing via the Airflow UI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org