You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/23 15:58:49 UTC
[GitHub] [airflow] chriskessel opened a new issue #16616: Can't use sftp connection defined in environment variables
chriskessel opened a new issue #16616:
URL: https://github.com/apache/airflow/issues/16616
I'm using the S3Hook, passing it a connection id and for some reason it's not able to find it if that connection is defined as an environment variable.
```
sftp_hook = SFTPHook(ftp_conn_id=self.sftp_conn_id)
```
```
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/models/connection.py", line 379, in get_connection_from_secrets
raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `sftp_local` isn't defined
```
The connection is definitely there, I can see it if I use the CLI to `get connections sftp_local`
```
$ airflow connections get sftp_local
id | conn_id | conn_type | blah blah
None | sftp_local | sftp | blah blah
```
I'm using other environment variables for connections (AWS Athena/S3 connections) and they work, so I'm not sure if I'm doing something wrong or it's a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-866976596
Are you sure you have the env variable it defined in the worker where you run the tasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriskessel commented on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
chriskessel commented on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-867010138
I'm fairly sure, I kill the docker process, rm the container, and restart it (via docker-compose.yml) before each usage as I've been testing. I even bashed into the worker and looked at the env vars and it was there. Which was no surprise, that's why the airflow CLI found it.
I blew away the complete airflow docker setup, blew away the containers and images. Restarted everything...and now it works.
I really don't get it, I defined 3 different _CONN_ variables all at the same time, all 3 showed up on the worker's env vars, all 3 showed up in the CLI, only 2 worked....but it's working after that complete nuke, so yay?
Magic...closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriskessel closed issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
chriskessel closed issue #16616:
URL: https://github.com/apache/airflow/issues/16616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-866963860
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-867030030
Well. I stopped counting the times I was sure about something I did - about 20 years ago ;). It helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chriskessel edited a comment on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
chriskessel edited a comment on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-867010138
I'm fairly sure, I kill the docker process, rm the container, and restart it (via docker-compose.yml) before each usage as I've been testing. I even bashed into the worker and looked at the env vars and it was there. Which was no surprise, that's why the airflow CLI found it.
I blew away the complete airflow docker setup, blew away the containers and images, refetched the images. Restarted everything...and now it works.
I really don't get it, I defined 3 different _CONN_ variables all at the same time, all 3 showed up on the worker's env vars, all 3 showed up in the CLI, only 2 worked....but it's working after that complete nuke, so yay?
Magic...closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16616: Can't use sftp connection defined in environment variables
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16616:
URL: https://github.com/apache/airflow/issues/16616#issuecomment-866976596
Are you sure you have the env variable it defined in the worker where you run the tasks? Maybe you need to restart it to pick the new env variable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org