You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/25 11:12:59 UTC
[GitHub] [airflow] potiuk opened a new pull request, #27912: Fixing the behaviours of SQL Hooks and Operators finally
potiuk opened a new pull request, #27912:
URL: https://github.com/apache/airflow/pull/27912
This is the more bold fix to common.sql and related providers. It implements more comprehensive tests and describes the intended behaviours in much more explicit way, also it simplifies the notion of "scalar" behaviour by introducing a new method to check on when scalar return is expected.
Comprehensive suite of tests have been added (the original tests were converted to Pytest test and parameterized was used to test all the combinations of parameters and returned values.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032392734
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -246,40 +270,48 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the ALL SQL expressions if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
+
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list: Iterable[str] = self.split_sql_string(sql)
else:
- sql = [sql]
+ sql_list = [sql] if sql.strip() else []
Review Comment:
I added strip() check here to cover the case when sql is empty string (see tests for empty).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032651745
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -234,7 +266,42 @@ def run(
"""
Runs a command or a list of commands. Pass a list of sql
statements to the sql parameter to get them to execute
- sequentially
+ sequentially.
+
+ The method will return either single query results (typically list of rows) or list of those results
+ where each element in the list are results of one of the queries (typically list of list of rows :D)
+
+ For compatibility reasons, the behaviour of the DBAPIHook is somewhat confusing.
+ In cases, when multiple queries are run, the return values will be an iterable (list) of results
+ - one for each query. However, in certain cases, when single query is run - the results will be just
+ the results of that query without wrapping the results in a list.
+
+ The cases when single query results are returned without wrapping them in a list are when:
+
+ a) sql is string and last_statement is True (regardless what split_statement value is)
+ b) sql is string and split_statement is False
+
+ In all other cases, the results are wrapped in a list, even if there is only one statement to process:
+
+ a) always when sql is iterable (regardless what last_statement value is)
+ b) when sql is string, split_statement is True and last_statement is False
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327913024
All looks good :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032572626
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
Review Comment:
Aaaaa. Htat would explain it !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327368060
cc: @alexott @kazanzhy - I have two important doubts about processing of `;` (and sql stripping in general).
1) (Less important if it works) Stripping ";"
I followed original implementations I think and I left I think intact original behviour of stripping in both implementationations.
Look at the parameterized tests - and the "DBApiHook" behaves now differently than Databricks Hook in this regard - the Databricks hook always splits the `;`
https://github.com/apache/airflow/pull/27912/files#diff-db2c1495ba766a1bad71474f36caeb3e8b4228522571e3229c1c64cede967027R99
Where the standard DBApiHook does not:
https://github.com/apache/airflow/pull/27912/files#diff-718d368660d07715b383dd4d94344bc41b835189da39bb95fcecaa3a423e6e47R110
2) The behaviour of cursor and connection opening/closing is different in both cases:
* DBAPiHook opens connection, cursors - and iterates over multiple queries with the same cursor (runs execute command for each) https://github.com/apache/airflow/pull/27912/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R288
* Databricks Hook opens connection, but then creates a cursor for each sql statement separately:
https://github.com/apache/airflow/pull/27912/files#diff-f1c71c8b869a687043b306f047f592e3638b3d74d9ec3c972360cf6c9a7956a9R182
3) I think we miss commiting after last statement in Databricks Hook. The DBApi hook does it https://github.com/apache/airflow/pull/27912/files#diff-6e1b2f961cb951d05d66d2d814ef5f6d8f8bf8f43c40fb5d40e27a031fed8dd7R307 but the DAtabricks one does not.
When autocommit is off, I think and no explicit commit given , then the transaction will be rolled back automatically
https://peps.python.org/pep-0249/#Connection.close
> Note that closing a connection without committing the changes first will cause an implicit rollback to be performed.
I am no sure what was the reasoning behind those differences - but from what I see those differences now between those two "run" methods. Question is - whether we have good reasons to behave differently for those cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
vincbeck commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327734237
Thanks @potiuk for doing that! I'll definitely take a look today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032648381
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -234,7 +266,42 @@ def run(
"""
Runs a command or a list of commands. Pass a list of sql
statements to the sql parameter to get them to execute
- sequentially
+ sequentially.
+
+ The method will return either single query results (typically list of rows) or list of those results
+ where each element in the list are results of one of the queries (typically list of list of rows :D)
+
+ For compatibility reasons, the behaviour of the DBAPIHook is somewhat confusing.
+ In cases, when multiple queries are run, the return values will be an iterable (list) of results
+ - one for each query. However, in certain cases, when single query is run - the results will be just
+ the results of that query without wrapping the results in a list.
+
+ The cases when single query results are returned without wrapping them in a list are when:
+
+ a) sql is string and last_statement is True (regardless what split_statement value is)
+ b) sql is string and split_statement is False
+
+ In all other cases, the results are wrapped in a list, even if there is only one statement to process:
+
+ a) always when sql is iterable (regardless what last_statement value is)
+ b) when sql is string, split_statement is True and last_statement is False
Review Comment:
Ah yeah. Mental shortcut. "Iterable of string statements" now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032360615
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -225,54 +225,37 @@ def __init__(
self.split_statements = split_statements
self.return_last = return_last
- @overload
- def _process_output(
- self, results: Any, description: Sequence[Sequence] | None, scalar_results: Literal[True]
- ) -> Any:
- pass
-
- @overload
- def _process_output(
- self, results: list[Any], description: Sequence[Sequence] | None, scalar_results: Literal[False]
- ) -> Any:
- pass
-
- def _process_output(
- self, results: Any | list[Any], description: Sequence[Sequence] | None, scalar_results: bool
- ) -> Any:
+ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequence] | None]) -> list[Any]:
"""
- Can be overridden by the subclass in case some extra processing is needed.
+ Processes output before it is returned by the operator.
+
+ It can be overridden by the subclass in case some extra processing is needed.
The "process_output" method can override the returned output - augmenting or processing the
output as needed - the output returned will be returned as execute return value and if
do_xcom_push is set to True, it will be set as XCom returned
:param results: results in the form of list of rows.
- :param description: as returned by ``cur.description`` in the Python DBAPI
- :param scalar_results: True if result is single scalar value rather than list of rows
+ :param descriptions: list of descriptions returned by ``cur.description`` in the Python DBAPI
"""
return results
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
- if self.do_xcom_push:
Review Comment:
No need to separate those two cases.
Word of explanation. Contrary to the original implementation here
We do not care whether "do_xcom_push" is set or not or not.
Whether we have scalar return value in case of hook does not matter - we can safely pass "return_last" to the hook because when do_xcom_push is False, the return value is anyhow ignored. So ¯\_(ツ)_/¯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032371554
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
@alexott Is there a good reason why there is a new connection open for every SQL statemetnt?
This construct (closing connection after each statement) causes a weird behaviour for autocommmit - basically - there is no way to commit anything manually because explicit rollback is executed after every statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032468739
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
I see - so we kinda accept that "autocommit off" is not really usable for now ? I thin that's fine, but maybe worth documenting it ? Because right now I do not really see any scenario when autocommit: off would be useful - if you "read" the data - autocommit is not used, but when you write the data, there is no way to commit the changes currently there is no way to do it when autocommit is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327739314
I’ll test when providers are prepared as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327841666
BTW. I REALLY ❤️ Pytest style of tests vs. the "classic" unit.TestCase.
Partucularrly fixtures - when you get a grasp of them, parameterization and the integration of those running tests within IDEs are just great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032354157
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -264,22 +291,26 @@ def run(
with closing(conn.cursor()) as cur:
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
self._run_command(cur, sql_statement, parameters)
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
Review Comment:
ok. I solved it much better now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032360615
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -225,54 +225,37 @@ def __init__(
self.split_statements = split_statements
self.return_last = return_last
- @overload
- def _process_output(
- self, results: Any, description: Sequence[Sequence] | None, scalar_results: Literal[True]
- ) -> Any:
- pass
-
- @overload
- def _process_output(
- self, results: list[Any], description: Sequence[Sequence] | None, scalar_results: Literal[False]
- ) -> Any:
- pass
-
- def _process_output(
- self, results: Any | list[Any], description: Sequence[Sequence] | None, scalar_results: bool
- ) -> Any:
+ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequence] | None]) -> list[Any]:
"""
- Can be overridden by the subclass in case some extra processing is needed.
+ Processes output before it is returned by the operator.
+
+ It can be overridden by the subclass in case some extra processing is needed.
The "process_output" method can override the returned output - augmenting or processing the
output as needed - the output returned will be returned as execute return value and if
do_xcom_push is set to True, it will be set as XCom returned
:param results: results in the form of list of rows.
- :param description: as returned by ``cur.description`` in the Python DBAPI
- :param scalar_results: True if result is single scalar value rather than list of rows
+ :param descriptions: list of descriptions returned by ``cur.description`` in the Python DBAPI
"""
return results
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
- if self.do_xcom_push:
Review Comment:
No need to separate those two cases.
Word of explanation. Contrary to the original implementation here
We do not care whether "do_xcom_push" is set or not or not.
Whether we have scalar return value in case of hook does not matter - we can safely pass "return_last" to the hook because when do_xcom_push is False, the return value is anyhow ignored. So `¯\_(ツ)_/¯`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032420592
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -225,54 +225,37 @@ def __init__(
self.split_statements = split_statements
self.return_last = return_last
- @overload
- def _process_output(
- self, results: Any, description: Sequence[Sequence] | None, scalar_results: Literal[True]
- ) -> Any:
- pass
-
- @overload
- def _process_output(
- self, results: list[Any], description: Sequence[Sequence] | None, scalar_results: Literal[False]
- ) -> Any:
- pass
-
- def _process_output(
- self, results: Any | list[Any], description: Sequence[Sequence] | None, scalar_results: bool
- ) -> Any:
+ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequence] | None]) -> list[Any]:
"""
- Can be overridden by the subclass in case some extra processing is needed.
+ Processes output before it is returned by the operator.
+
+ It can be overridden by the subclass in case some extra processing is needed.
The "process_output" method can override the returned output - augmenting or processing the
output as needed - the output returned will be returned as execute return value and if
do_xcom_push is set to True, it will be set as XCom returned
:param results: results in the form of list of rows.
- :param description: as returned by ``cur.description`` in the Python DBAPI
- :param scalar_results: True if result is single scalar value rather than list of rows
+ :param descriptions: list of descriptions returned by ``cur.description`` in the Python DBAPI
"""
return results
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
- if self.do_xcom_push:
Review Comment:
I only left `handler = None` if there is no `do_xcom_push`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032471088
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -225,54 +230,37 @@ def __init__(
self.split_statements = split_statements
self.return_last = return_last
- @overload
- def _process_output(
- self, results: Any, description: Sequence[Sequence] | None, scalar_results: Literal[True]
- ) -> Any:
- pass
-
- @overload
- def _process_output(
- self, results: list[Any], description: Sequence[Sequence] | None, scalar_results: Literal[False]
- ) -> Any:
- pass
-
- def _process_output(
- self, results: Any | list[Any], description: Sequence[Sequence] | None, scalar_results: bool
- ) -> Any:
+ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequence] | None]) -> list[Any]:
"""
- Can be overridden by the subclass in case some extra processing is needed.
+ Processes output before it is returned by the operator.
+
+ It can be overridden by the subclass in case some extra processing is needed.
The "process_output" method can override the returned output - augmenting or processing the
output as needed - the output returned will be returned as execute return value and if
- do_xcom_push is set to True, it will be set as XCom returned
+ do_xcom_push is set to True, it will be set as XCom returned.
:param results: results in the form of list of rows.
- :param description: as returned by ``cur.description`` in the Python DBAPI
- :param scalar_results: True if result is single scalar value rather than list of rows
+ :param descriptions: list of descriptions returned by ``cur.description`` in the Python DBAPI
"""
return results
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
- if self.do_xcom_push:
- output = hook.run(
- sql=self.sql,
- autocommit=self.autocommit,
- parameters=self.parameters,
- handler=self.handler,
- split_statements=self.split_statements,
- return_last=self.return_last,
- )
- else:
- output = hook.run(
- sql=self.sql,
- autocommit=self.autocommit,
- parameters=self.parameters,
- split_statements=self.split_statements,
- )
-
- return self._process_output(output, hook.last_description, hook.scalar_return_last)
+ output = hook.run(
+ sql=self.sql,
+ autocommit=self.autocommit,
+ parameters=self.parameters,
+ handler=self.handler if self.do_xcom_push else None,
+ split_statements=self.split_statements,
+ return_last=self.return_last,
+ )
+ if has_scalar_return_value(self.sql, self.return_last, self.split_statements):
+ # For simplicity, we pass always list as input to _process_output, regardless if
+ # scalar is going to be returned, and we return the first element of the list in this case
+ # from the list returned by _process_output
+ return self._process_output([output], hook.descriptions)[0]
Review Comment:
Because in this case it will be always one-element array. But yeah. I can change it back to -1. Both are a little confusing though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032623905
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -234,7 +266,42 @@ def run(
"""
Runs a command or a list of commands. Pass a list of sql
statements to the sql parameter to get them to execute
- sequentially
+ sequentially.
+
+ The method will return either single query results (typically list of rows) or list of those results
+ where each element in the list are results of one of the queries (typically list of list of rows :D)
+
+ For compatibility reasons, the behaviour of the DBAPIHook is somewhat confusing.
+ In cases, when multiple queries are run, the return values will be an iterable (list) of results
+ - one for each query. However, in certain cases, when single query is run - the results will be just
+ the results of that query without wrapping the results in a list.
+
+ The cases when single query results are returned without wrapping them in a list are when:
+
+ a) sql is string and last_statement is True (regardless what split_statement value is)
+ b) sql is string and split_statement is False
+
+ In all other cases, the results are wrapped in a list, even if there is only one statement to process:
+
+ a) always when sql is iterable (regardless what last_statement value is)
+ b) when sql is string, split_statement is True and last_statement is False
Review Comment:
I'm not sure I understand a?
string is also iterable. so a always happens thus b will never be checked?
Also, can you clarify the handling for when user is doing:
`SQLExecuteQueryOperator(..., sql=["SELECT 1"])`
vs
`SQLExecuteQueryOperator(..., sql="SELECT 1")`
In that case the are equivalent but from the description i'm not sure if handled the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032331484
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -264,22 +291,26 @@ def run(
with closing(conn.cursor()) as cur:
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
self._run_command(cur, sql_statement, parameters)
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
Review Comment:
We do not need to append value - only last is used anyway when scalar return value is used.
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -264,22 +291,26 @@ def run(
with closing(conn.cursor()) as cur:
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
self._run_command(cur, sql_statement, parameters)
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
Review Comment:
We do not need to append results - only last is used anyway when scalar return value is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327692059
Once the tests passes (and hopefully positive feedback) @ephraimbuddy @dstandish @uranusj @ashb or others - I would appreciate approavals before the weekend so I cancel and re-start the release vote for providers - to be on time for 2.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032371554
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
@alexott Is there a good reason why there is a new connection open for every SQL statemetnt?
This construct (closing connection after each statement) causes a weird behaviour for autocommmit off. Basically - there is no way to commit anything manually because explicit rollback is executed after every statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032463169
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -30,6 +30,25 @@
from airflow.version import version
+def has_scalar_return_value(sql: str | Iterable[str], return_last: bool, split_statements: bool):
+ """
+ Determines when scalar value should be returned.
+
+ Scalar value should be returned when:
Review Comment:
I was thinking about it - I think that terminology is confusing a bit. When we think about `scalar` we think about one value. But execution of query always return an iterator of rows (IIRc).
So at the end we have following possibilities:
* a list of query results (list of lists of rows) - when we have multiple queries
* a single query result (list of rows) - when we have one query or when we use `return_last`
maybe rename it to something like `single_result` or something like that ?
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -225,54 +230,37 @@ def __init__(
self.split_statements = split_statements
self.return_last = return_last
- @overload
- def _process_output(
- self, results: Any, description: Sequence[Sequence] | None, scalar_results: Literal[True]
- ) -> Any:
- pass
-
- @overload
- def _process_output(
- self, results: list[Any], description: Sequence[Sequence] | None, scalar_results: Literal[False]
- ) -> Any:
- pass
-
- def _process_output(
- self, results: Any | list[Any], description: Sequence[Sequence] | None, scalar_results: bool
- ) -> Any:
+ def _process_output(self, results: list[Any], descriptions: list[Sequence[Sequence] | None]) -> list[Any]:
"""
- Can be overridden by the subclass in case some extra processing is needed.
+ Processes output before it is returned by the operator.
+
+ It can be overridden by the subclass in case some extra processing is needed.
The "process_output" method can override the returned output - augmenting or processing the
output as needed - the output returned will be returned as execute return value and if
- do_xcom_push is set to True, it will be set as XCom returned
+ do_xcom_push is set to True, it will be set as XCom returned.
:param results: results in the form of list of rows.
- :param description: as returned by ``cur.description`` in the Python DBAPI
- :param scalar_results: True if result is single scalar value rather than list of rows
+ :param descriptions: list of descriptions returned by ``cur.description`` in the Python DBAPI
"""
return results
def execute(self, context):
self.log.info("Executing: %s", self.sql)
hook = self.get_db_hook()
- if self.do_xcom_push:
- output = hook.run(
- sql=self.sql,
- autocommit=self.autocommit,
- parameters=self.parameters,
- handler=self.handler,
- split_statements=self.split_statements,
- return_last=self.return_last,
- )
- else:
- output = hook.run(
- sql=self.sql,
- autocommit=self.autocommit,
- parameters=self.parameters,
- split_statements=self.split_statements,
- )
-
- return self._process_output(output, hook.last_description, hook.scalar_return_last)
+ output = hook.run(
+ sql=self.sql,
+ autocommit=self.autocommit,
+ parameters=self.parameters,
+ handler=self.handler if self.do_xcom_push else None,
+ split_statements=self.split_statements,
+ return_last=self.return_last,
+ )
+ if has_scalar_return_value(self.sql, self.return_last, self.split_statements):
+ # For simplicity, we pass always list as input to _process_output, regardless if
+ # scalar is going to be returned, and we return the first element of the list in this case
+ # from the list returned by _process_output
+ return self._process_output([output], hook.descriptions)[0]
Review Comment:
why we return first element? Usually we're interested in the last result (original behavior of DBSQL operator)
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
self._sql_conn = None
if handler is None:
return None
- elif self.scalar_return_last:
- return results[-1]
+ if has_scalar_return_value(sql, return_last, split_statements):
+ return results[0]
Review Comment:
Same here - we were always returning results of the last query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032457693
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
Review Comment:
There is no commit in Databricks SQL (yet)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032765656
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -190,11 +189,17 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
Executes SQL code in a specific database
:param sql: the SQL code or string pointing to a template file to be executed (templated).
File must have a '.sql' extensions.
+
+ When implementing a specific Operator, you can also implement `_process_output` method in the
Review Comment:
Likely :). I think it will work anyway in docstring (we'll see in the API docs) but yeah - we can fix it as follow-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032337056
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -264,22 +291,26 @@ def run(
with closing(conn.cursor()) as cur:
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
self._run_command(cur, sql_statement, parameters)
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
Review Comment:
I have a better idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032492519
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -30,6 +30,25 @@
from airflow.version import version
+def has_scalar_return_value(sql: str | Iterable[str], return_last: bool, split_statements: bool):
+ """
+ Determines when scalar value should be returned.
+
+ Scalar value should be returned when:
Review Comment:
I changed it everywhere to `return_single_query_results`. Note singular `query` but plural `results` :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327656422
OK. There was some permission issue - but I think this one shoudl be ready to go. I added some descriptions and explanations about the behaviours and :crossed_fingers: it will pass all the docs and tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327660251
Let me do a checkout in my environment and test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327738702
> I tested `DatabricksSqlOperator` with this patch, and all integration tests are passing (query with and without output, multiple queries + output, etc.).
Cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032763444
##########
airflow/providers/common/sql/operators/sql.py:
##########
@@ -190,11 +189,17 @@ class SQLExecuteQueryOperator(BaseSQLOperator):
Executes SQL code in a specific database
:param sql: the SQL code or string pointing to a template file to be executed (templated).
File must have a '.sql' extensions.
+
+ When implementing a specific Operator, you can also implement `_process_output` method in the
Review Comment:
should we use double backquotes in docstrings?
##########
airflow/providers/common/sql/CHANGELOG.rst:
##########
@@ -32,7 +32,9 @@ This release fixes a few errors that were introduced in common.sql operator whil
* ``_process_output`` method in ``SQLExecuteQueryOperator`` has now consistent semantics and typing, it
can also modify the returned (and stored in XCom) values in the operators that derive from the
``SQLExecuteQueryOperator``).
-* last description of the cursor whether to return scalar values are now stored in DBApiHook
+* descriptions of all returned results are stored as descriptions property in the DBApiHook
Review Comment:
(could be done later) maybe it would be a bit easier to read if `descriptions` and `last_description` are formatted as code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BillCM commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
BillCM commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1329496371
Interesting note: the 3.3.0 DatabricksSQLOperator had a parameter for control writing to xcom. It appears to be removed in 4.0.0. This is causing the dump of a lot of PII into XCOM for my use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1329507639
@BillCM `do_xcom_push` should be still supported as it's argument to the `BaseOperator` - can you check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032364140
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
Review Comment:
@alexott - do we miss commit here when autocommit is false ? This is what DBApiHook does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1328016558
BTW. Release of rc3 is coming .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327344602
cc: @alexott
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032371554
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
@alexott Is there a good reason why there is a new connection open for every SQL statemetnt?
This causes a weird behaviour for autocommmit - basically - there is no way to commit anything manually because explicit rollback is executed after every statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032457693
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
Review Comment:
There is no commit in Databricks SQL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032459844
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
Yes, this is done to support Azure Active Directory authentication - when we have long-running queries, AAD token may expire, so next queries will fail. To mitigate this, DBSQL hook checks if the AAD token is expiring, and renewing it. This is necessary right now as DBSQL connector for Python doesn't have built-in AAD auth support - this will come together with unified Python SDK next year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032395728
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
Review Comment:
I added strip_sql_string here to be consistent with the approach where no split_statements is set. It was inconsistent. Apparently it's ok for Databricks to run a single statement without ";" so we should be ok doing so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032392734
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -246,40 +270,48 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the ALL SQL expressions if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
+
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list: Iterable[str] = self.split_sql_string(sql)
else:
- sql = [sql]
+ sql_list = [sql] if sql.strip() else []
Review Comment:
I added strip() here to cover the case when sql is empty string (see tests for empty).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
alexott commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327727631
I tested `DatabricksSqlOperator` with this patch, and all integration tests are passing (query with and without output, multiple queries + output, etc.).
Huge thank you @potiuk for fixing it on generic level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032765479
##########
airflow/providers/common/sql/CHANGELOG.rst:
##########
@@ -32,7 +32,9 @@ This release fixes a few errors that were introduced in common.sql operator whil
* ``_process_output`` method in ``SQLExecuteQueryOperator`` has now consistent semantics and typing, it
can also modify the returned (and stored in XCom) values in the operators that derive from the
``SQLExecuteQueryOperator``).
-* last description of the cursor whether to return scalar values are now stored in DBApiHook
+* descriptions of all returned results are stored as descriptions property in the DBApiHook
Review Comment:
Yep. We can update it later :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327993893
It's holiday in us and I'm traveling, won't be able to review until next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032468739
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
Review Comment:
I see - so we kinda accept that "autocommit off" is not really usable for now ? I think that's fine, but maybe worth documenting it ? Because right now I do not really see any scenario when autocommit: off would be useful - if you "read" the data - autocommit is not used, but when you write the data, there is no way to commit the changes currently there is no way to do it when autocommit is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032470425
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -30,6 +30,25 @@
from airflow.version import version
+def has_scalar_return_value(sql: str | Iterable[str], return_last: bool, split_statements: bool):
+ """
+ Determines when scalar value should be returned.
+
+ Scalar value should be returned when:
Review Comment:
Yeah. I do not like `scalar` name in this context too - it is super confusing (this is actually that caused my changes in #27854 to go completely wrong.
Let me see if I can change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327673722
BTW. @vincbeck - re our recent discussions - this was not the problem of breaking compatibility with past relases, so it is not entirely the case we talked about in https://github.com/apache/airflow/pull/27854#issuecomment-1325568521 but I did end up with adding about a million tests to both common.sql provider and datbricks provider.
Maybe you want to take a look as well? Another pair of eyes would be handy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327674049
(or so it feels it's a million).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #27912:
URL: https://github.com/apache/airflow/pull/27912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032394307
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -246,40 +270,48 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the ALL SQL expressions if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
+
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list: Iterable[str] = self.split_sql_string(sql)
else:
- sql = [sql]
+ sql_list = [sql] if sql.strip() else []
+ else:
+ sql_list = sql
- if sql:
- self.log.debug("Executing following statements against DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
-
+ _last_result = None
with closing(self.get_conn()) as conn:
if self.supports_autocommit:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
self._run_command(cur, sql_statement, parameters)
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ _last_result = result
+ _last_description = cur.description
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
# If autocommit was set to False or db does not support autocommit, we do a manual commit.
if not self.get_autocommit(conn):
conn.commit()
if handler is None:
return None
- elif self.scalar_return_last:
- return results[-1]
+ if has_scalar_return_value(sql, return_last, split_statements):
+ self.descriptions = [_last_description]
Review Comment:
The `self.descriptions` is always an array - no matter if scalar return is used.
The assumption here is that if scalar is expected - you can use "last_decription" property. In all other cases you can use descriptions array .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032396667
##########
tests/providers/databricks/operators/test_databricks_sql.py:
##########
@@ -112,198 +271,6 @@ def test_exec_write_file(self, db_mock_class):
parameters=None,
handler=fetch_all_handler,
autocommit=False,
- return_last=True,
- split_statements=False,
- )
-
-
-class TestDatabricksSqlCopyIntoOperator(unittest.TestCase):
Review Comment:
Moved those tests to separate _copy module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327838202
> So much work on that PR! Nothing to say on my side, very solid! Thanks for adding all these tests (on top of the other changes), it adds a lot of robustness!
Indeed. I added already few small changes/clarifications after the initial push and adding all tests and it felt soooooooo reassuring to see green checkmark for all tests after I did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27912:
URL: https://github.com/apache/airflow/pull/27912#discussion_r1032372319
##########
airflow/providers/databricks/hooks/databricks_sql.py:
##########
@@ -163,38 +163,43 @@ def run(
:param return_last: Whether to return result for only last statement or for all after split
:return: return only result of the LAST SQL expression if handler was provided.
"""
- self.scalar_return_last = isinstance(sql, str) and return_last
+ self.descriptions = []
if isinstance(sql, str):
if split_statements:
- sql = self.split_sql_string(sql)
+ sql_list = [self.strip_sql_string(s) for s in self.split_sql_string(sql)]
else:
- sql = [self.strip_sql_string(sql)]
+ sql_list = [self.strip_sql_string(sql)]
+ else:
+ sql_list = [self.strip_sql_string(s) for s in sql]
- if sql:
- self.log.debug("Executing following statements against Databricks DB: %s", list(sql))
+ if sql_list:
+ self.log.debug("Executing following statements against Databricks DB: %s", sql_list)
else:
raise ValueError("List of SQL statements is empty")
results = []
- for sql_statement in sql:
+ for sql_statement in sql_list:
# when using AAD tokens, it could expire if previous query run longer than token lifetime
with closing(self.get_conn()) as conn:
self.set_autocommit(conn, autocommit)
with closing(conn.cursor()) as cur:
self._run_command(cur, sql_statement, parameters)
-
if handler is not None:
result = handler(cur)
- results.append(result)
- self.last_description = cur.description
+ if has_scalar_return_value(sql, return_last, split_statements):
+ results = [result]
+ self.descriptions = [cur.description]
+ else:
+ results.append(result)
+ self.descriptions.append(cur.description)
Review Comment:
It's even worse now when I look at that - it seems that "autocommit = off" makes basically no sense because there is implicit rollback after every statement when connection is closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #27912: Fixing the behaviours of SQL Hooks and Operators finally
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #27912:
URL: https://github.com/apache/airflow/pull/27912#issuecomment-1327693264
I'll take a look when I can find time at a computer, this is definitely not one to review on a phone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org