You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dacamposol (via GitHub)" <gi...@apache.org> on 2025/03/19 16:57:36 UTC
[I] Problem with Rotation of Fernet Key [airflow]
dacamposol opened a new issue, #47963:
URL: https://github.com/apache/airflow/issues/47963
### Official Helm Chart version
1.15.0 (latest released)
### Apache Airflow version
2.9.3
### Kubernetes Version
1.31.4
### Helm Chart configuration
```yaml
createUserJob:
applyCustomEnv: false
useHelmHooks: false
dags:
gitSync:
branch: main
credentialsSecret: airflow-git-credentials
enabled: true
repo: https://github.com/dacamposol/kg-infra.git
subPath: dags
extraEnv: |
- name: AIRFLOW__API__AUTH_BACKENDS
value: 'airflow.api.auth.backend.basic_auth'
migrateDatabaseJob:
applyCustomEnv: false
jobAnnotations:
'argocd.argoproj.io/hook': Sync
useHelmHooks: false
useStandardNaming: true
```
### Docker Image customizations
_No response_
### What happened
> [!IMPORTANT]
> This is an instance managed via ArgoCD.
1. I created a connection from type HTTP directly from the Web Server UI.
2. I modified a field in the Argo Application in regards of adding the `.extraEnv` to allow REST API calls.
3. I accessed again to the Web Server UI.
4. Cannot open the **Connections** section (all others work fine).
Error in the `webserver` Pod:
```bash
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/mapper.py", line 3702, in _event_on_load
instrumenting_mapper._reconstructor(state.obj())
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 213, in on_db_load
if self.password:
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 606, in __get__
retval = self.descriptor.__get__(instance, owner)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 340, in get_password
return fernet.decrypt(bytes(self._password, "utf-8")).decode()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cryptography/fernet.py", line 211, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
```
When I go to my Argo CD deployment, I can indeed see how for the Airflow application, the `airflow-fernet-key` got recreated at the same time than the sync happened.
I am assuming the Secret got recreated and somehow the rotation process failed, so the **Connections** are now encrypted with the old key, and thus cannot be accessed anymore.
### What you think should happen instead
_No response_
### How to reproduce
1. Have an Airflow Cluster managed by Argo CD.
2. Create a connection in that cluster.
3. Modify settings in the Application, and wait for Sync.
4. Attempt to access again to the Connections in the aforementioned cluster.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-4752273460
Is this issue still relevant when working with the up-to-date new helm chart? If so, please provide reproduction steps.
Otherwise, this issue will be auto-closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-2875550327
If so, the solution would be to take the fernet key created the first time and put it directly in the values file.
That way ArgoCD will use the same one every time and not create a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "stephsi (via GitHub)" <gi...@apache.org>.
stephsi commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-2875468903
Your recommendation (@nevcohen ) regarding selective sync does not work.
The fernet key is created by a pre-install helm hook. ArgoCD executes this hook on every sync (see below)
**Some context for the problem in general:**
When upgrading airflow using helm, one uses `helm upgrade` and thus the pre-install hook is not executed and the fernet key is not replaced.
This is different to how ArgoCD works ([doc link](https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-hooks))
> Argo CD cannot know if it is running a first-time "install" or an "upgrade" - every operation is a "sync'. This means that, by default, apps that have pre-install and pre-upgrade will have those hooks run at the same time.
This means ArgoCD triggers the pre-install hook everytime as you described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "arivictor (via GitHub)" <gi...@apache.org>.
arivictor commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-3470718666
Was there a resolution to this? As I am experiencing the same issue (maybe due to lack of Airflow setup experience).
1. We deploy once via Helm on ArgoCD
2. We make modifications to values.yaml
3. We push the changes and ArgoCD attempts to resync
4. Variables and Connections are now inaccessible via the UI
To resolve, we have to delete the whole deployment and reinstall. Fortunately our application is stateless so wiping airflow isn't too big of a deal but doesn't seem right.
We have our fernet key hardcoded and this is still the issue. Do we need to set a concrete secret instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "Miretpl (via GitHub)" <gi...@apache.org>.
Miretpl commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-4759889721
Related #68265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-2737399646
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-4884292187
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-2737764132
As you said, the secret that contain the fernet key is initialized at every sync, try to add [this](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#selective-sync) in your argocd app.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Problem with Rotation of Fernet Key [airflow]
Posted by "arivictor (via GitHub)" <gi...@apache.org>.
arivictor commented on issue #47963:
URL: https://github.com/apache/airflow/issues/47963#issuecomment-3475673973
I found a solution that seems to work:
**values.yaml**
```yaml
airflow
fernetKeySecretName: "voltflow-fernet-key"
extraSecrets:
"voltflow-fernet-key":
type: "Opaque"
data: |
fernet-key: "<base-64 encoded fernet-key>"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org