You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/07 13:28:16 UTC
[GitHub] [airflow] mik-laj opened a new pull request #20106: Support insecure mode in SnowflakeHook
mik-laj opened a new pull request #20106:
URL: https://github.com/apache/airflow/pull/20106
Close: https://github.com/apache/airflow/issues/19797
CC: @mattpolzin @harishkrao
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764408021
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -210,19 +236,38 @@ def _get_conn_params(self) -> Dict[str, Optional[str]]:
def get_uri(self) -> str:
"""Override DbApiHook get_uri method for get_sqlalchemy_engine()"""
- conn_config = self._get_conn_params()
+ conn_params = self._get_conn_params()
+ return self._conn_params_to_sqlalchemy_uri(conn_params)
+
+ def _conn_params_to_sqlalchemy_uri(self, conn_params):
uri = (
'snowflake://{user}:{password}@{account}.{region}/{database}/{schema}'
'?warehouse={warehouse}&role={role}&authenticator={authenticator}'
)
- return uri.format(**conn_config)
+ return uri.format(**conn_params)
def get_conn(self) -> SnowflakeConnection:
"""Returns a snowflake.connection object"""
conn_config = self._get_conn_params()
conn = connector.connect(**conn_config)
return conn
+ def get_sqlalchemy_engine(self, engine_kwargs=None):
+ """
+ Get an sqlalchemy_engine object.
+
+ :param engine_kwargs: Kwargs used in :func:`~sqlalchemy.create_engine`.
+ :return: the created engine.
+ """
+ if engine_kwargs is None:
+ engine_kwargs = {}
Review comment:
```suggestion
engine_kwargs = engine_kwargs or {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764567672
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
I changed the behavior and for an invalid value an exception is now raised
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764735984
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
`to_boolean` looks interesting. Updated and. pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764734341
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
Pushed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764407162
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
I'm not sure about the return - what it means it terms of boolean value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764570799
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
Also we have `airflow.utils.strings.to_boolean()`. Its behaviour is different from the one you have here, but might be more expected since the function is used in other parts of Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r767456710
##########
File path: docs/apache-airflow-providers-snowflake/connections/snowflake.rst
##########
@@ -63,6 +63,9 @@ Extra (optional)
* ``private_key_file``: Specify the path to the private key file.
* ``session_parameters``: Specify `session level parameters
<https://docs.snowflake.com/en/user-guide/python-connector-example.html#setting-session-parameters>`_
+ * ``insecure_mode``: Turns off OCSP certificate checks
Review comment:
```suggestion
* ``insecure_mode``: Turn off OCSP certificate checks
```
This is consistent to the tense used by other bullet points here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #20106:
URL: https://github.com/apache/airflow/pull/20106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#issuecomment-988732259
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764538586
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
If it can be converted to a bool value then a conversation is performed, but if an incorrect value is passed it is passed unchanged and then the layer below can handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764407670
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -210,19 +236,38 @@ def _get_conn_params(self) -> Dict[str, Optional[str]]:
def get_uri(self) -> str:
"""Override DbApiHook get_uri method for get_sqlalchemy_engine()"""
- conn_config = self._get_conn_params()
+ conn_params = self._get_conn_params()
+ return self._conn_params_to_sqlalchemy_uri(conn_params)
+
+ def _conn_params_to_sqlalchemy_uri(self, conn_params):
Review comment:
```suggestion
def _conn_params_to_sqlalchemy_uri(self, conn_params: Dict):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764395777
##########
File path: docs/apache-airflow-providers-snowflake/connections/snowflake.rst
##########
@@ -63,11 +63,14 @@ Extra (optional)
* ``private_key_file``: Specify the path to the private key file.
* ``session_parameters``: Specify `session level parameters
<https://docs.snowflake.com/en/user-guide/python-connector-example.html#setting-session-parameters>`_
- * ``aws_access_key_id``: Specify your aws S3 access key for use with the S3ToSnowflakeOperator.
- * ``aws_secret_access_key``: Specify your aws S3 secret access key for use with the S3ToSnowflakeOperator.
+ * ``insecure_mode``: Turns off OCSP certificate checks
+ For details, see: `How To: Turn Off OCSP Checking in Snowflake Client Drivers - Snowflake Community
+ <https://community.snowflake.com/s/article/How-to-turn-off-OCSP-checking-in-Snowflake-client-drivers>`__.
+ * ``aws_access_key_id``: Specify your aws S3 access key for use with the :class:`~airflow.providers.snowflake.transfers.s3_to_snowflake.S3ToSnowflakeOperator`.
+ * ``aws_secret_access_key``: Specify your aws S3 secret access key for use with the :class:`~airflow.providers.snowflake.transfers.s3_to_snowflake.S3ToSnowflakeOperator`.
When specifying the connection in environment variable you should specify
-it using URI syntax.
+it using :doc:`URI syntax <apache-airflow:connection/uri-syntax>`.
Review comment:
```suggestion
it using :ref:`URI syntax <apache-airflow:connection/uri-syntax>`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#issuecomment-988817901
Tests are failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #20106: Support insecure mode in SnowflakeHook
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #20106:
URL: https://github.com/apache/airflow/pull/20106#discussion_r764570006
##########
File path: airflow/providers/snowflake/hooks/snowflake.py
##########
@@ -25,10 +25,22 @@
from snowflake import connector
from snowflake.connector import DictCursor, SnowflakeConnection
from snowflake.connector.util_text import split_statements
+from sqlalchemy import create_engine
from airflow.hooks.dbapi import DbApiHook
+def _boolify(value):
+ if isinstance(value, bool):
+ return value
+ if isinstance(value, str):
+ if value.lower() == 'false':
+ return False
+ elif value.lower() == 'true':
+ return True
+ return value
Review comment:
Did you not push the change? The implementation here does not raise, from what I can tell.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org