You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/07 19:00:33 UTC
[GitHub] [airflow] lewismc opened a new issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
lewismc opened a new issue #16313:
URL: https://github.com/apache/airflow/issues/16313
**Apache Airflow version**: 2.1.0 (Helm chart) from main branch 5c7d758e24595c485553b0449583ff238114d47d
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-12T01:08:32Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6+k3s1", GitCommit:"8d0432824a9fd9474b67138b7630c33f285d332f", GitTreeState:"clean", BuildDate:"2021-04-16T19:04:44Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}```
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): macOS Catalina 10.15.7
- **Kernel** (e.g. `uname -a`): Darwin MT-207576 19.6.0 Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64 x86_64
- **Install tools**: helm version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}
- **Others**:
**What happened**:
After installaing Airflow and port forwarding from kubectl, when I navigate to the **connections** dropdown menu, I get the following trace
```
Ooops!
Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.
Python version: 3.6.13
Airflow version: 2.1.0
Node: airflow-webserver-67d745bff9-5sm7w
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/airflow/.local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/home/airflow/.local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/views.py", line 551, in list
widgets = self._list()
File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/baseviews.py", line 1134, in _list
page_size=page_size,
File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/baseviews.py", line 1033, in _get_list_widget
page_size=page_size,
File "/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py", line 435, in query
query_results = query.all()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3373, in all
return list(self)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
cursor.close()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
rows = [proc(row) for row in fetch]
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
rows = [proc(row) for row in fetch]
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 601, in _instance
state.manager.dispatch.load(state, context)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/event/attr.py", line 322, in __call__
fn(*args, **kw)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/mapper.py", line 3397, in _event_on_load
instrumenting_mapper._reconstructor(state.obj())
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py", line 150, in on_db_load
if self.password:
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 365, in __get__
retval = self.descriptor.__get__(instance, owner)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py", line 235, in get_password
return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
File "/home/airflow/.local/lib/python3.6/site-packages/cryptography/fernet.py", line 194, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
```
**What you expected to happen**:
I expected to see the Connections pane.
**How to reproduce it**:
```
git clone airflow && cd airflow/chart
helm install airflow . -n airflow-test
kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow-test
```
Navigate to http://localhost:8080/connection/list/ and the trace appears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-856182930
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Gurulhu commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
Gurulhu commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-860768457
I think I found what the issue might be:
Inspecting the code it fails at [password decryption](https://github.com/apache/airflow/blob/943292b4e0c494f023c86d648289b1f23ccb0ee9/airflow/models/connection.py#L237) so the issue wasn't about having no token.
My environment wasn't properly clean between releases and the postgres RDS got init'd with a random-generated fernet key A, and the current release went up with another random generated fernet key B without ever rotating the key in the RDS, as it's out of the helm delete scope.
Wiping the RDS before spinning the chart up fixed it. (Test db, no useful data was harmed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Gurulhu commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
Gurulhu commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-860709591
@jedcunningham I'm not the OP but I'm encountering the exact same issue.
I'm running the official helm chart (http://airflow.apache.org/docs/helm-chart/stable/index.html) on Amazon EKS, and my permissions are kinda iffy, but are mostly working by now.
`echo $AIRFLOW__CORE__FERNET_KEY` on webserver matches the fenet key in secrets, and it's a valid fernet key (to be 100% sure I generated one and hardcodded it on the values and it still gives the same error.
I'm still debugging it and will update this issue if I find anything relevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-856309532
I'm not able to reproduce this. Do you get a base64 string when you run the "echo Fernet Key" command `helm install` prints? e.g:
```
echo Fernet Key: $(kubectl get secret --namespace airflow-test airflow-fernet-key -o jsonpath="{.data.fernet-key}" | base64 --decode)
```
What are you using for K8s? What version of Helm do you have? I assume a new release name doesn't help, e.g. "airflow" -> "test"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-856310140
If you exec into the webserver pod and run `echo $AIRFLOW__CORE__FERNET_KEY`, does it match the "echo Fernet Key" command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #16313:
URL: https://github.com/apache/airflow/issues/16313
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-884574032
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-860788656
Cool, glad you found it!
@lewismc, that was why I asked about a new release name actually, as with the default values the chart keeps the postgres PVC around and can cause this issue. Can you confirm if you have the same underlying issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #16313: cryptography.fernet.InvalidToken when attempting to access connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #16313:
URL: https://github.com/apache/airflow/issues/16313#issuecomment-880288890
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org