You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/14 16:15:29 UTC
[GitHub] [airflow] potiuk opened a new pull request, #25713: Fix SQL split string to include `;-less` statements
potiuk opened a new pull request, #25713:
URL: https://github.com/apache/airflow/pull/25713
There was a bug in an incoming change to common-sql provider
introduced in #23971 where `;-less` statements were removed
when "split_statements" flag was used. Since this flag is used
by default in Databricks statement, it introduced backwards
incompatible change.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945316074
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
I wonder what you think about the FP style:
```
statements: List[str] = list(filter(None, map(lambda x: x.rstrip(';'), splits)))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945316074
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
I wonder what you think about the FP style:
```
statements: List[str] = list(filter(None, map(lambda x: x.rstrip(';'), splits)))
```
Seems `sqlparse` already removes spaces from the queries, so probably `.strip()` is not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25713:
URL: https://github.com/apache/airflow/pull/25713#issuecomment-1214409115
cc: @kazanzhy @alexott
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25713:
URL: https://github.com/apache/airflow/pull/25713#issuecomment-1214410890
@kazanzhy @alexott - I chose a little different approach than proposed in https://github.com/apache/airflow/issues/25640#issuecomment-1213002702 - after adding unit tests. I generaly filter out all empty statements during the split - that will even allow empty comments for example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945332362
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
```suggestion
statements = list(
filter(None, (s.rstrip(';') if s.endswith(';') else s for s in splits))
)
```
The `strip()` in the `else` part isn’t consistent. Alternatively, `strip()` should be done for both cases:
```python
statements = (s.rstrip(';') if s.endswith(';') else s for s in splits)
statements = (s.strip() for s in statements)
return [s for s in statements if s]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945316074
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
I wonder what you think about the FP style:
```suggestion
statements: List[str] = list(filter(None, map(lambda x: x.rstrip(';'), splits)))
```
Seems `sqlparse` already removes spaces from the queries, so probably `.strip()` is not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945316074
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
I wonder what you think about the FP style:
```
statements: List[str] = list(filter(None, map(lambda x: x.rstrip(';'), splits)))
```
Seems `sqlparse` already removes spaces from the queries, so probably `.strip()` is not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25713:
URL: https://github.com/apache/airflow/pull/25713#issuecomment-1214668898
I think this one should be ready to review :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945316074
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
I wonder what you think about the FP style:
```
statements: List[str] = list(filter(None, map(lambda x: x.strip().rstrip(';'), splits)))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945334454
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
> Sorry for being offline. I tried to work on #25259 finally.
No worries :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25713:
URL: https://github.com/apache/airflow/pull/25713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on PR #25713:
URL: https://github.com/apache/airflow/pull/25713#issuecomment-1214413551
Sorry for being offline. I tried to work on #25259 finally.
Initially, I took this code from the drill provider where it was like:
```
sql = sqlparse.split(sqlparse.format(self.sql, strip_comments=True))
no_term_sql = [s[:-1] for s in sql if s[-1] == ';']
```
which is the like current `[s.rstrip(';') for s in splits if s.endswith(';')]`
I tested it with different queries and even with comments and I see that I really missed the query like `'SELECT * FROM table;;;;;'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945333543
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
Ah yeah. Lost it in the last change :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25713: Fix SQL split string to include `;-less` statements
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25713:
URL: https://github.com/apache/airflow/pull/25713#discussion_r945334358
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -244,7 +244,9 @@ def split_sql_string(sql: str) -> List[str]:
:return: list of individual expressions
"""
splits = sqlparse.split(sqlparse.format(sql, strip_comments=True))
- statements = [s.rstrip(';') for s in splits if s.endswith(';')]
+ statements: List[str] = list(
+ filter(None, [s.rstrip(';') if s.endswith(';') else s.strip() for s in splits])
+ )
Review Comment:
Fixed and added test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org