You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (Jira)" <ji...@apache.org> on 2019/12/06 17:39:00 UTC
[jira] [Updated] (AIRFLOW-6192) Stop creating Hook from
SFTPSensor.__init__
[ https://issues.apache.org/jira/browse/AIRFLOW-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kaxil Naik updated AIRFLOW-6192:
--------------------------------
Description:
From Slack (https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1575647500146200):
When I create a task based upon the SFTPSensor and then parse the Python file, do I see two log entries coming from base_hook.py referencing the default sftp connection id? I also see the same log messages whenever the DAGs are refreshed.
{noformat}
airflow@0030a4ae49df:~$ python dags/sftp_sensor.py
[2019-12-06 15:45:16,440] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=9685
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-12-06 15:45:17,535] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
[2019-12-06 15:45:17,539] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
airflow@0030a4ae49df:~$ airflow list_dags
[2019-12-06 15:47:43,529] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=11223
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-12-06 15:47:44,631] {{__init__.py:51}} INFO - Using executor LocalExecutor
[2019-12-06 15:47:44,631] {{dagbag.py:92}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-12-06 15:47:48,998] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
[2019-12-06 15:47:49,005] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
{noformat}
This is on v1.10.6
was:
From Slack (https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1575647500146200):
When I create a task based upon the SFTPSensor and then parse the Python file, do I see two log entries coming from base_hook.py referencing the default sftp connection id? I also see the same log messages whenever the DAGs are refreshed.
airflow@0030a4ae49df:~$ python dags/sftp_sensor.py
[2019-12-06 15:45:16,440] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=9685
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-12-06 15:45:17,535] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
[2019-12-06 15:45:17,539] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
airflow@0030a4ae49df:~$ airflow list_dags
[2019-12-06 15:47:43,529] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=11223
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-12-06 15:47:44,631] {{__init__.py:51}} INFO - Using executor LocalExecutor
[2019-12-06 15:47:44,631] {{dagbag.py:92}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-12-06 15:47:48,998] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
[2019-12-06 15:47:49,005] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
This is on v1.10.6
> Stop creating Hook from SFTPSensor.__init__
> -------------------------------------------
>
> Key: AIRFLOW-6192
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6192
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
> Affects Versions: 1.10.6
> Reporter: Kaxil Naik
> Assignee: Kaxil Naik
> Priority: Minor
> Fix For: 1.10.7
>
>
> From Slack (https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1575647500146200):
> When I create a task based upon the SFTPSensor and then parse the Python file, do I see two log entries coming from base_hook.py referencing the default sftp connection id? I also see the same log messages whenever the DAGs are refreshed.
> {noformat}
> airflow@0030a4ae49df:~$ python dags/sftp_sensor.py
> [2019-12-06 15:45:16,440] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=9685
> /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
> """)
> [2019-12-06 15:45:17,535] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
> [2019-12-06 15:45:17,539] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
> airflow@0030a4ae49df:~$ airflow list_dags
> [2019-12-06 15:47:43,529] {{settings.py:252}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=11223
> /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
> """)
> [2019-12-06 15:47:44,631] {{__init__.py:51}} INFO - Using executor LocalExecutor
> [2019-12-06 15:47:44,631] {{dagbag.py:92}} INFO - Filling up the DagBag from /usr/local/airflow/dags
> [2019-12-06 15:47:48,998] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
> [2019-12-06 15:47:49,005] {{base_hook.py:84}} INFO - Using connection to: id: sftp_default. Host: localhost, Port: 22, Schema: None, Login: airflow, Password: None, extra: {'key_file': '~/.ssh/id_rsa', 'no_host_key_check': True}
> {noformat}
> This is on v1.10.6
--
This message was sent by Atlassian Jira
(v8.3.4#803005)