You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/02 06:51:44 UTC
[GitHub] [airflow] Stormhand opened a new issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Stormhand opened a new issue #16769:
URL: https://github.com/apache/airflow/issues/16769
Hello,
I have an Airflow 2.1 deployment on kubernetes which is integrated with Hashicorp Vault as secure backend. I found out that the integration will be fully completed only when we get rid of two more keys which I am obligated to apply as secrets:
- gitSync ssh file
- Fernet key
The easiest way would be if they could be loaded the same way the hashicorp vault provider client is loading its token from a file. For example i have the ssh key and the fernet key in vault as secrets which are injected by the Vault agent as pod annotations:
```
vault.hashicorp.com/agent-inject-secret-airflow: "k8s-secrets/int/airflow"
vault.hashicorp.com/role: "airflow"
vault.hashicorp.com/agent-inject-template-airflow: |
{{- with secret "k8s-secrets/int/airflow" -}}
{{ .Data.vaulttoken -}}
{{- end }}
vault.hashicorp.com/agent-inject-secret-gitssh: "k8s-secrets/int/airflow"
vault.hashicorp.com/agent-inject-template-gitssh: |
{{- with secret "k8s-secrets/skube-int/airflow" -}}
{{ .Data.gitssh -}}
{{- end }}
vault.hashicorp.com/agent-inject-secret-fernetkey: "k8s-secrets/int/airflow"
vault.hashicorp.com/agent-inject-template-fernetkey: |
{{- with secret "k8s-secrets/skube-int/airflow" -}}
{{ .Data.fernetkey -}}
{{- end }}
```
Then in `/vault/secrets/` i have all the three files. The current configuration for GIT_SSH_KEY_FILE for Git-Sync SSH key is (in the _helpers.yaml file):
```
{{- if .Values.dags.gitSync.sshKeySecret }}
- name: GIT_SSH_KEY_FILE
value: "/etc/git-secret/ssh"
- name: GIT_SYNC_SSH
value: "true"
```
So there we might have another parameter **Values.dags.gitSync.sshKeySecretFile** which will point to the injected secret.
The Fernet Key however is a different story as its either written in the config, loaded from an ENV var or loaded from a secret:
```
# Hard Coded Airflow Envs
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
name: {{ template "fernet_key_secret" . }}
key: fernet-key
```
I am still not sure this could be implemented to read from a file. At first glance there are two solutions:
- Add an additional mechanism in the Airflow core to load the fernet from a file - this requires a lot of work i guess.
- Inject the key in the file as an export command:` export AIRFLOW__CORE__FERNET_KEY={{ .Data.fernetkey -}}` - this will probably require an additional script in the entrypoint of the container to execute the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-872981828
@Stormhand https://github.com/apache/airflow/issues/16684 is it related? In this ticket, we are waiting for more feedback, so your Vault Injector experience would be very valuable to us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #16769:
URL: https://github.com/apache/airflow/issues/16769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-872763480
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-895632220
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-873405426
How about we simply add an option "AIRFLOW__CORE__FERNET_KEY_FILE" in Airflow and read the key from there? this would be like a few line change. AS you might see from #16684, I do not feel particular warmth for the _CMD pattern. While flexible, it's also dangerous and might lead to promote bad behaviour and maybe just being able to specify file name for the FERNET_KEY is a nice, generic solution tha can solve this particular problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Stormhand commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
Stormhand commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-872996778
> @Stormhand #16684 is it related? In this ticket, we are waiting for more feedback, so your Vault Injector experience would be very valuable to us.
Interesting! I didnt know about `_CMD` and yes it can be used for the Fernet key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16769: Airflow Helm charts - better integration with Hashicorp Vault
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16769:
URL: https://github.com/apache/airflow/issues/16769#issuecomment-891411276
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org