You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "sebastianliebscher (via GitHub)" <gi...@apache.org> on 2023/11/14 11:53:26 UTC
[PR] chore: Cleanup function for stripping comments from SQL statements [superset]
sebastianliebscher opened a new pull request, #25980:
URL: https://github.com/apache/superset/pull/25980
### SUMMARY
- `sql_parse.strip_comments_from_sql()` needed to instantiate `ParsedQuery` just to strip comments from a raw SQL string. Now it is the other way around, `ParsedQuery.strip_comments()` uses `strip_comments_from_sql()`.
- dry up duplicate strip-comments-logic in `ParsedQuery` constructor, in `ParsedQuery.strip_comments()` and `ParsedQuery.is_<EXPLAIN|SHOW|SET>`
- make use of `sql_parse.strip_comments_from_sql()` in the codebase
- add additional tests to cover comments in SQL statements
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
- local unit / integration tests + GitHub CI
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #25980:
URL: https://github.com/apache/superset/pull/25980#issuecomment-2305464592
@sebastianliebscher this one never got a review, and I'm sorry about that... we're trying to get better about this. If you can give this a rebase, we'll try to help get it across the finish line (finally)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "sebastianliebscher (via GitHub)" <gi...@apache.org>.
sebastianliebscher commented on code in PR #25980:
URL: https://github.com/apache/superset/pull/25980#discussion_r1392482920
##########
tests/unit_tests/sql_parse_tests.py:
##########
@@ -1192,14 +1192,35 @@ def test_messy_breakdown_statements() -> None:
"""
SELECT 1;\t\n\n\n \t
\t\nSELECT 2;
-SELECT * FROM birth_names;;;
-SELECT * FROM birth_names LIMIT 1
+SELECT '--abc' FROM birth_names;;;
+SELECT * FROM birth_names LIMIT 1 --comment
"""
)
assert query.get_statements() == [
"SELECT 1",
"SELECT 2",
- "SELECT * FROM birth_names",
+ "SELECT '--abc' FROM birth_names",
+ "SELECT * FROM birth_names LIMIT 1 --comment",
+ ]
+
+
+def test_messy_breakdown_statements_with_comments() -> None:
+ """
+ Test the messy multiple statements are parsed correctly with 'strip_comments=True'.
+ """
+ query = ParsedQuery(
+ """
+SELECT 1;\t\n\n\n \t
+\t\nSELECT 2;
+SELECT '--abc' FROM birth_names;;;
+SELECT * FROM birth_names LIMIT 1 --comment
+""",
+ strip_comments=True,
Review Comment:
This test was useful during the refactor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "sebastianliebscher (via GitHub)" <gi...@apache.org>.
sebastianliebscher commented on code in PR #25980:
URL: https://github.com/apache/superset/pull/25980#discussion_r1392481604
##########
superset/sqllab/query_render.py:
##########
@@ -58,9 +58,9 @@ def render(self, execution_context: SqlJsonExecutionContext) -> str:
database=query_model.database, query=query_model
)
- parsed_query = ParsedQuery(query_model.sql, strip_comments=True)
rendered_query = sql_template_processor.process_template(
- parsed_query.stripped(), **execution_context.template_params
+ strip_comments_from_sql(query_model.sql),
+ **execution_context.template_params,
Review Comment:
Does not need `ParsedQuery` just to `return sqlparse_format(string)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "sebastianliebscher (via GitHub)" <gi...@apache.org>.
sebastianliebscher commented on code in PR #25980:
URL: https://github.com/apache/superset/pull/25980#discussion_r1392485165
##########
superset/sql_parse.py:
##########
@@ -152,15 +154,14 @@ def get_cte_remainder_query(sql: str) -> tuple[Optional[str], str]:
def strip_comments_from_sql(statement: str) -> str:
"""
- Strips comments from a SQL statement, does a simple test first
- to avoid always instantiating the expensive ParsedQuery constructor
+ Strips comments from an SQL statement.
- This is useful for engines that don't support comments
+ This is useful for engines that don't support comments.
:param statement: A string with the SQL statement
:return: SQL statement without comments
"""
- return ParsedQuery(statement).strip_comments() if "--" in statement else statement
+ return sqlparse.format(statement.strip(" \t\r\n;"), strip_comments=True)
Review Comment:
Now every call makes use of `statement.strip(" \t\r\n;")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #25980:
URL: https://github.com/apache/superset/pull/25980#issuecomment-2305465398
I'll put it in "draft" mode until then... feel free to change it back at any time :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas commented on PR #25980:
URL: https://github.com/apache/superset/pull/25980#issuecomment-2702148382
This PR hasn't seen any activity/events in roughly seven months. I'll reluctantly close it, but would love to see it reopened/revisited if anyone here has interest or bandwidth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] chore: Cleanup function for stripping comments from SQL statements [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas closed pull request #25980: chore: Cleanup function for stripping comments from SQL statements
URL: https://github.com/apache/superset/pull/25980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org