You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/22 23:05:43 UTC
[GitHub] [airflow] mgovindasamy opened a new issue, #28547: Snowflake Connection - Failing
mgovindasamy opened a new issue, #28547:
URL: https://github.com/apache/airflow/issues/28547
### Apache Airflow version
2.5.0
### What happened
Snowflake Connection Cursor is failing all the time with SIGSEGV, the same code just calling as pyscript is working fine
Airflow Output:
[2022-12-22, 22:38:41 UTC] {connection.py:280} INFO - Snowflake Connector for Python Version: 2.9.0, Python Version: 3.8.16, Platform: macOS-13.0.1-x86_64-i386-64bit
[2022-12-22, 22:38:41 UTC] {connection.py:975} INFO - THIS CONNECTION IS IN INSECURE MODE. IT MEANS THE CERTIFICATE WILL BE VALIDATED BUT THE CERTIFICATE REVOCATION STATUS WILL NOT BE CHECKED.
[2022-12-22, 22:38:41 UTC] {connection.py:983} INFO - Setting use_openssl_only mode to False
[2022-12-22, 22:38:41 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-12-22, 22:38:41 UTC] {taskinstance.py:2582} INFO - 0 downstream tasks scheduled from follow-on schedule check
pyScript Output:
[2022-12-22 14:35:42,689] {connection.py:280} INFO - Snowflake Connector for Python Version: 2.9.0, Python Version: 3.8.16, Platform: macOS-13.0.1-x86_64-i386-64bit
[2022-12-22 14:35:42,689] {connection.py:975} INFO - THIS CONNECTION IS IN INSECURE MODE. IT MEANS THE CERTIFICATE WILL BE VALIDATED BUT THE CERTIFICATE REVOCATION STATUS WILL NOT BE CHECKED.
[2022-12-22 14:35:42,689] {connection.py:983} INFO - Setting use_openssl_only mode to False
[2022-12-22 14:35:43,214] {ssl_wrap_socket.py:99} INFO - THIS CONNECTION IS IN INSECURE MODE. IT MEANS THE CERTIFICATE WILL BE VALIDATED BUT THE CERTIFICATE REVOCATION STATUS WILL NOT BE CHECKED.
[2022-12-22 14:35:43,541] {cursor.py:727} INFO - query: [SELECT current_version()]
[2022-12-22 14:35:43,646] {cursor.py:740} INFO - query execution done
[2022-12-22 14:35:43,646] {cursor.py:878} INFO - Number of results in first chunk: 1
6.41.2
[2022-12-22 14:35:43,647] {connection.py:581} INFO - closed
[2022-12-22 14:35:43,735] {connection.py:584} INFO - No async queries seem to be running, deleting session
I have setup the connection parameter with ,insecure_mode=True to turn off OCSP in snowflake, I assume the socket wrapper is not getting called when it triggers
### What you think should happen instead
import logging
from datetime import datetime, timedelta
import airflow
import os
import sys
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
#from airflow.contrib.hooks.snowflake_hook import SnowflakeHook
#from airflow.contrib.operators.snowflake_operator import SnowflakeOperator
# Operators
from airflow.operators.empty import EmptyOperator
from airflow.operators.python import PythonOperator
#from _scproxy import _get_proxy_settings
#import botocore.vendored.requests, urllib.request
import snowflake.connector
# Gets the version
def sfConnectCursor():
ctx = snowflake.connector.connect(
user='USER',
password='password',
account='sfaccount',insecure_mode=True,
)
cs = ctx.cursor()
try:
cs.execute("SELECT current_version()")
one_row = cs.fetchone()
print(one_row[0])
finally:
cs.close()
ctx.close()
#def sfConnectCursor():
# print("Welcome back")
#def _noProxy():
# if sys.platform == 'darwin':
# botocore.vendored.requests.utils.proxy_bypass = urllib.request.proxy_bypass_environment
# botocore.vendored.requests.utils.getproxies = urllib.request.getproxies_environment
# urllib.request.proxy_bypass = urllib.request.proxy_bypass_environment
# urllib.request.getproxies = urllib.request.getproxies_environment
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
args = {"owner": "Airflow", "start_date": datetime(2021,3,22,17,15)}
snowflake_connector = DAG(dag_id="snowflake_connector_dag", default_args=args, schedule_interval=None,catchup=False)
query1 = [
"""select 1;""",
"""show tables in database BROADCAST_SESSION_METRICS;""",
]
#def count1(**context):
# _get_proxy_settings()
# _noProxy()
# os.environ["no_proxy"]="*"
# os.environ['export PYTHONFAULTHANDLER="true"']
# dwh_hook = SnowflakeHook(snowflake_conn_id="snowflake_conn")
# result = dwh_hook.get_first("select current_version")
# logging.info("Number of rows in - %s", result[0])
#with dag:
# query1_exec = SnowflakeOperator(
# task_id="snowfalke_task1",
# sql=query1,
# snowflake_conn_id="snowflake_conn",
# )
#count_query = PythonOperator(task_id="count_query", python_callable=sfConnectCursor)
#query1_exec >> count_query
# Creating first task
start_task = EmptyOperator(task_id='start_task', dag=snowflake_connector)
# Creating second task
hello_world_task = PythonOperator(task_id='sf_task', python_callable=sfConnectCursor, dag=snowflake_connector)
# Creating third task
end_task = EmptyOperator(task_id='end_task', dag=snowflake_connector)
# Set the order of execution of tasks.
start_task >> hello_world_task >> end_task
#count_query
#sfConnectCursor()
### How to reproduce
Attached is the dag, which is reproduced every time.
Running on MAC
### Operating System
Mac-OS
### Versions of Apache Airflow Providers
$ pip freeze|grep apache-airflow-providers
apache-airflow-providers-common-sql==1.3.1
apache-airflow-providers-ftp==3.2.0
apache-airflow-providers-http==4.1.0
apache-airflow-providers-imap==3.1.0
apache-airflow-providers-snowflake==4.0.2
apache-airflow-providers-sqlite==3.3.1
(venv) usf8ffc22b527f:connector mgovindasamy$
### Deployment
Other
### Deployment details
_No response_
### Anything else
I think the problem is opening the socket when it is calling the airflow
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28547: Snowflake Connection - Failing
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28547:
URL: https://github.com/apache/airflow/issues/28547#issuecomment-1363423525
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed issue #28547: Snowflake Connection - Failing
Posted by GitBox <gi...@apache.org>.
Taragolis closed issue #28547: Snowflake Connection - Failing
URL: https://github.com/apache/airflow/issues/28547
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28547: Snowflake Connection - Failing
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28547:
URL: https://github.com/apache/airflow/issues/28547#issuecomment-1363468965
> [2022-12-22, 22:38:41 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV
This could reference to known issue with fork process in MacOS, there are quite a few different solutions you could find in discussions:
https://github.com/apache/airflow/discussions?discussions_q=Negsignal.SIGSEGV
We also add some header if we found Segmentation Fault errors in task with some hint how to debug:
https://github.com/apache/airflow/pull/27381/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org