You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/09 22:17:26 UTC
[GitHub] [airflow] hpatel-higi opened a new issue #18127: Fernet InvalidToken
hpatel-higi opened a new issue #18127:
URL: https://github.com/apache/airflow/issues/18127
### Apache Airflow version
2.1.3 (latest released)
### Operating System
Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-hive==2.0.2
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-jdbc==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-snowflake==2.1.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.1.0
### Deployment
Docker-Compose
### Deployment details
version: "2.1"
services:
postgres:
image: postgres:13.4
container_name: "airflow_postgres_qa"
environment:
- POSTGRES_HOST=postgres
- POSTGRES_PORT:5432
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
webserver:
image: airflow_docker
container_name: "airflow_webserver_qa"
entrypoint: "/entrypoint.sh"
command: "webserver"
restart: always
depends_on:
- postgres
ports:
- "9003:9003"
# - "2222:2222"
# - "8080:80"
environment:
EXECUTOR: "Local"
AIRFLOW__WEBSERVER__BASE_URL: "http://localhost:9003"
volumes:
- ../../keys:/usr/local/airflow/keys
- ../pipes:/usr/local/airflow/configs/snowpipe
- ../../support_files:/usr/local/airflow/support_files
- ../../dags:/usr/local/airflow/dags
- ../../sql:/usr/local/airflow/sql
- ../access_control.json:/usr/local/airflow/configs/access_control/access_control.json
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
### What happened
Adding or deleting connections using CLI in entrypoint.sh file throws error:
```
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/connection_command.py", line 229, in connections_delete
to_delete = session.query(Connection).filter(Connection.conn_id == args.conn_id).one()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
cursor.close()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
rows = [proc(row) for row in fetch]
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 601, in _instance
state.manager.dispatch.load(state, context)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 322, in __call__
fn(*args, **kw)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3397, in _event_on_load
instrumenting_mapper._reconstructor(state.obj())
File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 150, in on_db_load
if self.password:
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 365, in __get__
retval = self.descriptor.__get__(instance, owner)
File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 235, in get_password
return fernet.decrypt(bytes(self._password, 'utf-8')).decode()
File "/usr/local/lib/python3.9/site-packages/cryptography/fernet.py", line 194, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
```
### What you expected to happen
Expected the connections to be deleted and readded again.
### How to reproduce
- Dont set FERNET_KEY value in airflow.cfg file
- Set FERNET_KEY from inside entrypoint.sh file
- Run container first time
- First time it should work fine
- Run container again and it will throw Invalid Token errors
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-916484838
In case you do not set the fernet key, Airflow will generate a new key, which means that if you restart the container, you may lose access to the key. You should explicitly define the key in docker-compose.yaml. You can also set an empty string to disable encryption.
Here is example:
https://github.com/apache/airflow/blob/2fd3f27ebd2a4f695968fb7cc5d119e087820928/docs/apache-airflow/start/docker-compose.yaml#L58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hpatel-higi commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
hpatel-higi commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-916486900
In my entrypoint.sh file i have this line of code to generate the fernet key
```: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"```
Are you saying that i should not generate it in the entrypoint.sh but instead just provide it in the docker-compose file??
This worked like this in 1.10.8. We're upgrading to 2.1.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hpatel-higi commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
hpatel-higi commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-917070868
Here is a screenshot of the connections page from the airflow 1.10.8 website. You can see the connections are marked as Encrypted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hpatel-higi commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
hpatel-higi commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-916540395
I actually have the crypto package installed in my airflow 1.10.8 docker container.
here is the pip freeze:
```
alembic==1.7.1
amqp==2.6.1
apache-airflow==1.10.8
apispec==1.3.3
argcomplete==1.12.3
asn1crypto==1.4.0
attrs==19.3.0
azure-common==1.1.27
azure-core==1.18.0
azure-data-tables==12.1.0
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
Babel==2.9.1
bcrypt==3.2.0
billiard==3.6.4.0
boto3==1.13.26
botocore==1.16.26
cached-property==1.5.2
cachelib==0.3.0
cachetools==4.2.2
cattrs==0.9.2
celery==4.4.7
certifi==2020.12.5
cffi==1.13.2
chardet==3.0.4
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.4
colorlog==4.0.2
configparser==3.5.3
contextvars==2.4
croniter==0.3.37
cryptography==2.9.2
dask==2021.3.0
dataclasses==0.8
datadog==0.42.0
defusedxml==0.7.1
dill==0.3.4
distributed==1.28.1
dnspython==2.1.0
docutils==0.15.2
email-validator==1.1.3
Flask==1.1.4
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.4
Flask-Babel==1.0.0
Flask-Bcrypt==0.7.1
Flask-Caching==1.3.3
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OAuthlib==0.9.6
Flask-OpenID==1.3.0
Flask-SQLAlchemy==2.5.1
flask-swagger==0.2.13
Flask-WTF==0.14.3
flower==0.9.7
funcsigs==1.0.2
furl==2.1.2
future==0.16.0
google-api-core==1.31.2
google-auth==1.35.0
googleapis-common-protos==1.53.0
graphviz==0.17
gunicorn==19.10.0
HeapDict==1.0.1
hmsclient==0.1.1
humanize==3.11.0
idna==2.9
ijson==2.6.1
immutables==0.16
importlib-metadata==4.8.1
importlib-resources==5.2.2
iso8601==0.1.16
isodate==0.6.0
itsdangerous==1.1.0
JayDeBeApi==1.2.3
Jinja2==2.10.3
jmespath==0.10.0
JPype1==1.3.0
json-merge-patch==0.2
jsonschema==3.2.0
kombu==4.6.11
lazy-object-proxy==1.6.0
lockfile==0.12.2
Mako==1.1.5
Markdown==2.6.11
MarkupSafe==2.0.1
marshmallow==2.21.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.23.1
msgpack==1.0.2
msrest==0.6.21
mysqlclient==1.3.14
natsort==7.1.1
ndg-httpsclient==0.5.1
numpy==1.19.5
oauthlib==2.1.0
opencensus==0.7.13
opencensus-context==0.1.2
opencensus-ext-azure==1.0.8
orderedmultidict==1.0.1
oscrypto==1.2.1
packaging==21.0
pandas==0.25.3
pendulum==1.4.4
prison==0.2.1
prometheus-client==0.8.0
protobuf==3.17.3
psutil==5.8.0
psycopg2-binary==2.9.1
pyarrow==0.17.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodomex==3.10.1
Pygments==2.10.0
PyHive==0.6.4
PyJWT==1.7.1
pymssql==2.1.4
pyodbc==4.0.30
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.18.0
python-daemon==2.1.2
python-dateutil==2.8.2
python3-openid==3.2.0
pytz==2020.1
pytzdata==2020.1
PyYAML==5.4.1
redis==2.10.6
requests==2.23.0
requests-oauthlib==1.1.0
rsa==4.7.2
s3transfer==0.3.7
setproctitle==1.2.2
six==1.16.0
slackclient==1.3.2
snowflake-connector-python==2.2.6
snowflake-ingest==1.0.2
snowflake-sqlalchemy==1.2.3
sortedcontainers==2.4.0
SQLAlchemy==1.3.15
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.37.8
tabulate==0.8.9
tblib==1.7.0
tenacity==4.12.0
termcolor==1.1.0
text-unidecode==1.2
thrift==0.13.0
toolz==0.11.1
tornado==5.1.1
typing==3.7.4.3
typing-extensions==3.10.0.2
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.25.11
vine==1.3.0
websocket-client==0.54.0
Werkzeug==0.16.0
WTForms==2.3.3
zict==2.0.0
zipp==3.5.0
zope.deprecation==4.4.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #18127:
URL: https://github.com/apache/airflow/issues/18127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-916503503
I don't understand why it worked in Airflow 1.10, but generating a new key every time the container is started will definitely not allow the data to be decrypted. You must use the same key for encryption and decryption.
You may not have had Fernet at all in Airflow 1.10 as it requires the installation of the crypto package. In Airflow 2.0, the fernet is turned on by default and you can turn it off if you want.
https://github.com/apache/airflow/blob/main/UPDATING.md#fernet-is-enabled-by-default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18127: Fernet InvalidToken
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18127:
URL: https://github.com/apache/airflow/issues/18127#issuecomment-916478979
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org