You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/28 21:18:05 UTC
[GitHub] [airflow] dstandish opened a new pull request, #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
dstandish opened a new pull request, #28635:
URL: https://github.com/apache/airflow/pull/28635
Some databases, such as snowflake, require you to split statements in order to submit multi-statement sql. For such databases, splitting is the natural default, and we should defer to the hook to control that.
cc @potiuk @eladkal @kazanzhy @pgagnon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059254233
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -252,13 +253,17 @@ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequen
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
+ if self.split_statements is not None:
+ extra_kwargs = {"split_statements": self.split_statements}
+ else:
+ extra_kwargs = {}
Review Comment:
thanks for the suggestion. current form is result of TP's suggestion. just gonna leave it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1058602568
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -26,6 +26,7 @@
from airflow.hooks.base import BaseHook
from airflow.models import BaseOperator, SkipMixin
from airflow.providers.common.sql.hooks.sql import DbApiHook, fetch_all_handler, return_single_query_results
+from airflow.utils.types import NOTSET
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059063811
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -198,7 +198,7 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
:param autocommit: (optional) if True, each command is automatically committed (default: False).
:param parameters: (optional) the parameters to render the SQL query with.
:param handler: (optional) the function that will be applied to the cursor (default: fetch_all_handler).
- :param split_statements: (optional) if split single SQL string into statements (default: False).
+ :param split_statements: (optional) if split single SQL string into statements (default: see hook.run).
Review Comment:
it's true it's true... was trying to stay on one line :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
pgagnon commented on PR #28635:
URL: https://github.com/apache/airflow/pull/28635#issuecomment-1366926674
> I don't think this breaks any backcompat because for most providers this is no change and, for snowflake , if you try to submit multistatement r.n. with defaults it will just fail
We should be very explicit about this in the release notes though if we merge this change (which I am in favor of), because it potentially alters DAG behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059237313
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -252,13 +253,16 @@ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequen
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
+ extra_kwargs = {}
+ if self.split_statements is not None:
+ extra_kwargs.update(split_statements=self.split_statements)
Review Comment:
```suggestion
if self.split_statements is not None:
extra_kwargs = {"split_statements": self.split_statements}
else:
extra_kwargs = {}
```
##########
airflow/providers/common/sql/operators/sql.pyi:
##########
@@ -75,7 +75,7 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
autocommit: bool = ...,
parameters: Union[Mapping, Iterable, None] = ...,
handler: Callable[[Any], Any] = ...,
- split_statements: bool = ...,
+ split_statements: Union[bool, None] = ...,
Review Comment:
```suggestion
split_statements: Optional[bool] = ...,
```
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -198,7 +198,8 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
:param autocommit: (optional) if True, each command is automatically committed (default: False).
:param parameters: (optional) the parameters to render the SQL query with.
:param handler: (optional) the function that will be applied to the cursor (default: fetch_all_handler).
- :param split_statements: (optional) if split single SQL string into statements (default: False).
+ :param split_statements: (optional) if split single SQL string into statements. By default, defers
+ to the default value in the ``run`` method in the configured hook.
Review Comment:
```suggestion
to the default value in the ``run`` method of the configured hook.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish merged pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish merged PR #28635:
URL: https://github.com/apache/airflow/pull/28635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1058741250
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -198,7 +198,7 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
:param autocommit: (optional) if True, each command is automatically committed (default: False).
:param parameters: (optional) the parameters to render the SQL query with.
:param handler: (optional) the function that will be applied to the cursor (default: fetch_all_handler).
- :param split_statements: (optional) if split single SQL string into statements (default: False).
+ :param split_statements: (optional) if split single SQL string into statements (default: see hook.run).
Review Comment:
What does *see hook.run* mean? Perhaps we should explain a bit more here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
pgagnon commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059251399
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -252,13 +253,17 @@ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequen
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
+ if self.split_statements is not None:
+ extra_kwargs = {"split_statements": self.split_statements}
+ else:
+ extra_kwargs = {}
Review Comment:
```suggestion
extra_kwargs = {"split_statements": self.split_statements} if self.split_statements else {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059115201
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -198,7 +198,7 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
:param autocommit: (optional) if True, each command is automatically committed (default: False).
:param parameters: (optional) the parameters to render the SQL query with.
:param handler: (optional) the function that will be applied to the cursor (default: fetch_all_handler).
- :param split_statements: (optional) if split single SQL string into statements (default: False).
+ :param split_statements: (optional) if split single SQL string into statements (default: see hook.run).
Review Comment:
ok @uranusjr , updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pgagnon commented on a diff in pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
pgagnon commented on code in PR #28635:
URL: https://github.com/apache/airflow/pull/28635#discussion_r1059251399
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -252,13 +253,17 @@ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequen
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
+ if self.split_statements is not None:
+ extra_kwargs = {"split_statements": self.split_statements}
+ else:
+ extra_kwargs = {}
Review Comment:
```suggestion
extra_kwargs = {"split_statements": self.split_statements} if self.split_statements else {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #28635: Defer to hook setting for split_statements in SQLExecuteQueryOperator
Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #28635:
URL: https://github.com/apache/airflow/pull/28635#issuecomment-1366922185
I don't think this breaks any backcompat because for most providers this is no change and, for snowflake , if you try to submit multistatement r.n. with defaults it will just fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org