You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/07/16 03:06:13 UTC
[GitHub] [superset] john-bodley opened a new pull request, #20729: fix: Ensure Presto database engine spec correctly handles Trino
john-bodley opened a new pull request, #20729:
URL: https://github.com/apache/superset/pull/20729
<!---
Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
Example:
fix(dashboard): load charts correctly
-->
### SUMMARY
This PR remedies a few issues with Trino—which extends the Presto database engine spec. Specifically at it relates to fetching extra metadata as part of the SQL Lab schema/table workflow.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
### TESTING INSTRUCTIONS
CI.
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dungdm93 commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1215081320
@villebro perhaps `latest_partition` issue is fixed in trinodb/trino-python-client#219. Waiting for next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r939048444
##########
superset/db_engine_specs/presto.py:
##########
@@ -947,20 +961,16 @@ def get_create_view(
:param schema: Schema name
:param table: Table (view) name
"""
- # pylint: disable=import-outside-toplevel
- from pyhive.exc import DatabaseError
engine = cls.get_engine(database, schema)
with closing(engine.raw_connection()) as conn:
cursor = conn.cursor()
sql = f"SHOW CREATE VIEW {schema}.{table}"
try:
cls.execute(cursor, sql)
-
- except DatabaseError: # not a VIEW
+ return cls.fetch_data(cursor, 1)[0][0]
+ except SupersetDBAPIDatabaseError: # not a VIEW
Review Comment:
Sadly because we're using a raw connection, i.e., outside of the SQLAlchemy realm, the error comes from the underlying DBAPI, i.e., it's wrapped by SQLAlchemy, and thus we need to define the `get_dbapi_exception_mapping` mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r922618432
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +906,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
+
+ if cols:
+ full_table_name = table_name
Review Comment:
Same logic as previous just indented under the `if cols:` statement.
Unlike Presto where `get_indexes` returns `[]` for a non-partition table, Trino returns `[{'name': 'partition', 'column_names': [], 'unique': False}]`. Rather than overriding the engine specific `normalize_indexes` method I though it would be more prudent to make this method more robust given there was already an expectation that there may be no columns associated with the index, i.e., a non-partitioned table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] codecov[bot] commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1186075798
# [Codecov](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#20729](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c7f04ee) into [master](https://codecov.io/gh/apache/superset/commit/e5e886739460c011a885a13b873665410045a19c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e5e8867) will **decrease** coverage by `15.17%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #20729 +/- ##
===========================================
- Coverage 66.36% 51.19% -15.18%
===========================================
Files 1754 1754
Lines 66671 66670 -1
Branches 7049 7049
===========================================
- Hits 44247 34131 -10116
- Misses 20627 30742 +10115
Partials 1797 1797
```
| Flag | Coverage Δ | |
|---|---|---|
| hive | `?` | |
| mysql | `?` | |
| postgres | `?` | |
| presto | `?` | |
| python | `50.41% <0.00%> (-31.30%)` | :arrow_down: |
| sqlite | `?` | |
| unit | `50.41% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [superset/db\_engine\_specs/presto.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3ByZXN0by5weQ==) | `26.55% <0.00%> (-62.48%)` | :arrow_down: |
| [superset/examples/css\_templates.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZXhhbXBsZXMvY3NzX3RlbXBsYXRlcy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/reports/notifications/email.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9ub3RpZmljYXRpb25zL2VtYWlsLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/reports/notifications/exceptions.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9ub3RpZmljYXRpb25zL2V4Y2VwdGlvbnMucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [superset/reports/commands/alert.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9hbGVydC5weQ==) | `0.00% <0.00%> (-96.60%)` | :arrow_down: |
| [superset/reports/notifications/base.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9ub3RpZmljYXRpb25zL2Jhc2UucHk=) | `0.00% <0.00%> (-95.84%)` | :arrow_down: |
| [superset/cli/test.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvY2xpL3Rlc3QucHk=) | `0.00% <0.00%> (-93.94%)` | :arrow_down: |
| [superset/reports/commands/execute.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9leGVjdXRlLnB5) | `0.00% <0.00%> (-91.61%)` | :arrow_down: |
| [superset/reports/commands/log\_prune.py](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9sb2dfcHJ1bmUucHk=) | `0.00% <0.00%> (-89.29%)` | :arrow_down: |
| ... and [305 more](https://codecov.io/gh/apache/superset/pull/20729/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [e5e8867...c7f04ee](https://codecov.io/gh/apache/superset/pull/20729?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dungdm93 commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1206166295
@john-bodley 2 projects are now different enough to have separated EngineSpec (at least in driver side).
You just save a few line of code by let `TrinoEngineSpec` inherit messy code from `PrestoEngineSpec` (and introducing bugs).
> at the moment there is a significant amount of the code that is the same
The significant amount of the code you said is used to patch PyHive driver. Trino in the other hand strictly follow DB-API 2, so default implementations from `BaseEngineSpec` are good enough.
Other utility functions like `estimate_statement_cost`, `query_cost_formatter` may the same. However, those func can be reused by move to an `utils.py` if u wanna do DRY.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dungdm93 commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1207125197
@john-bodley An other way to archive the DRY principle is using python mixin pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1206711938
@ktmud thanks for the review. I've actually had to update the logic since your approval per https://github.com/apache/superset/pull/20729#discussion_r939048444.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] villebro commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
villebro commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1209318314
@john-bodley I have to agree with @dungdm93 here - Over the years the Presto spec has caused me considerable grief due to the code being IMO messy and difficult to maintain. For that reason I've been very happy to see the work that @dungdm93 has done on the Trino spec, which IMO is of a great quality and easy follow and extend. So rather than double up on using the old Presto spec code, I'd rather see us gradually phasing it out.
WRT to compatibility of the drivers, I think they're fairly far away from each other right now. For instance, currently clicking on a table in SQL Lab with the recommended `trino` driver gives off a metadata fetching error (causes a 500 in the backend):
<img width="853" alt="image" src="https://user-images.githubusercontent.com/33317356/183643500-ebb7b70c-761b-4805-83ed-7209280b6f1b.png">
This is due to the `latest_partition` method in the Presto spec being incompatible with the `trino` driver. There are other similar issues; I'm opening a PR soon to fix query cancellation, which also works slightly differently, but there appear to be other differences, too.
If we really want to share code between the Presto and Trino specs I would recommend doing something similar to what we're doing for Postgres and Postgres-like databases, where we have the uncontroversial shared pieces in an abstract `PostgresBaseEngineSpec` which the actual implementations, like `PostgresEngineSpec`, `HanaEngineSpec`, `RedshiftEngineSpec` etc use. In this case, we could have a `PrestoBaseEngineSpec` into which we move the high quality shared code, and then leave the messy bits in the old `PrestoEngineSpec`, while gradually refactoring them into the shared `PrestoBaseEngineSpec`, all while adding proper and relevant tests for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r922618432
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +906,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
+
+ if cols:
+ full_table_name = table_name
Review Comment:
Same logic as previous just indented under the `if cols:` statement. Unlike Presto where `get_indexes` returns `[]` for a non-partition table, Trino returns `[{'name': 'partition', 'column_names': [], 'unique': False}]`. Rather than overriding the engine specific `normalize_indexes` method I though it would be more prudent to make this method more robust given there was already an expectation that there may be no columns associated with the index, i.e., a non-partitioned table.
##########
superset/db_engine_specs/presto.py:
##########
@@ -940,20 +942,19 @@ def get_create_view(
:param schema: Schema name
:param table: Table (view) name
"""
- # pylint: disable=import-outside-toplevel
- from pyhive.exc import DatabaseError
engine = cls.get_engine(database, schema)
with closing(engine.raw_connection()) as conn:
cursor = conn.cursor()
sql = f"SHOW CREATE VIEW {schema}.{table}"
try:
cls.execute(cursor, sql)
+ return cls.fetch_data(cursor, 1)[0][0]
Review Comment:
This should never have been a `pyhive.exc` exception to begin with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] ktmud commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
ktmud commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r939490686
##########
superset/db_engine_specs/trino.py:
##########
@@ -45,6 +46,15 @@ class TrinoEngineSpec(PrestoEngineSpec):
engine_aliases = {"trinonative"} # Required for backwards compatibility.
engine_name = "Trino"
+ @classmethod
+ def get_dbapi_exception_mapping(cls) -> Dict[Type[Exception], Type[Exception]]:
Review Comment:
Why is this a function instead of attribute? Do we expect the exceptions to map differently at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1205906240
@dungdm93 I think in the future when Presto and Trino further diverge it makes sense, however at the moment there is a significant amount of the code that is the same and I sense adhering to the DRY principle (where possible makes sense).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r938996859
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +907,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
- if not latest_parts:
- latest_parts = tuple([None] * len(col_names))
- metadata["partitions"] = {
- "cols": cols,
- "latest": dict(zip(col_names, latest_parts)),
- "partitionQuery": pql,
- }
+ if cols:
+ full_table_name = table_name
+ if schema_name and "." not in table_name:
+ full_table_name = "{}.{}".format(schema_name, table_name)
Review Comment:
@ktmud this code is unchanged, i.e., it's now nested under the `if cols:` section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dungdm93 commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1207132688
@john-bodley an other way to reusing code is the mixing pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r922628584
##########
tests/integration_tests/db_engine_specs/presto_tests.py:
##########
@@ -892,19 +892,6 @@ def test_get_create_view_exception(self):
)
schema = "schema"
table = "table"
- with self.assertRaises(Exception):
Review Comment:
I'm not entirely sure when/why a broad exception should be raised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley closed pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley closed pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
URL: https://github.com/apache/superset/pull/20729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] ktmud commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
ktmud commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r938351037
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +907,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
- if not latest_parts:
- latest_parts = tuple([None] * len(col_names))
- metadata["partitions"] = {
- "cols": cols,
- "latest": dict(zip(col_names, latest_parts)),
- "partitionQuery": pql,
- }
+ if cols:
+ full_table_name = table_name
+ if schema_name and "." not in table_name:
+ full_table_name = "{}.{}".format(schema_name, table_name)
Review Comment:
f-string?
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +907,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
- if not latest_parts:
- latest_parts = tuple([None] * len(col_names))
- metadata["partitions"] = {
- "cols": cols,
- "latest": dict(zip(col_names, latest_parts)),
- "partitionQuery": pql,
- }
+ if cols:
+ full_table_name = table_name
+ if schema_name and "." not in table_name:
+ full_table_name = "{}.{}".format(schema_name, table_name)
+ pql = cls._partition_query(full_table_name, database)
+ col_names, latest_parts = cls.latest_partition(
+ table_name, schema_name, database, show_first=True
+ )
+
+ if not latest_parts:
+ latest_parts = tuple([None] * len(col_names))
+ metadata["partitions"] = {
+ "cols": cols,
+ "latest": dict(zip(col_names, latest_parts)),
Review Comment:
I feel we should probably change the signature of `latest_partition()` itself to return this dict---would worth another PR.
For this code, it seems it can be simplified as:
```suggestion
"latest": {
col: latest_parts[i] if latest_parts else None
for i, col in enumerate(col_names)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on a diff in pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on code in PR #20729:
URL: https://github.com/apache/superset/pull/20729#discussion_r938996859
##########
superset/db_engine_specs/presto.py:
##########
@@ -906,21 +907,23 @@ def extra_table_metadata(
indexes = database.get_indexes(table_name, schema_name)
if indexes:
cols = indexes[0].get("column_names", [])
- full_table_name = table_name
- if schema_name and "." not in table_name:
- full_table_name = "{}.{}".format(schema_name, table_name)
- pql = cls._partition_query(full_table_name, database)
- col_names, latest_parts = cls.latest_partition(
- table_name, schema_name, database, show_first=True
- )
- if not latest_parts:
- latest_parts = tuple([None] * len(col_names))
- metadata["partitions"] = {
- "cols": cols,
- "latest": dict(zip(col_names, latest_parts)),
- "partitionQuery": pql,
- }
+ if cols:
+ full_table_name = table_name
+ if schema_name and "." not in table_name:
+ full_table_name = "{}.{}".format(schema_name, table_name)
Review Comment:
@ktmud this code is unchanged, i.e., it's now nested under the `if cols:` section and thus I would prefer not make changes to said code, at least in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] dungdm93 commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
dungdm93 commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1192728228
@john-bodley Trino using official driver, not PyHive as Presto.
This mean that a lot of function like `get_table_names`, `get_view_names`, `handle_cursor`, etc... implement naively in driver.
Do u think we should change it back to `TrinoEngineSpec(BaseEngineSpec)` to avoid inherit messy code from `PrestoEngineSpec` (and prevent bugs as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] john-bodley commented on pull request #20729: fix: Ensure Presto database engine spec correctly handles Trino
Posted by GitBox <gi...@apache.org>.
john-bodley commented on PR #20729:
URL: https://github.com/apache/superset/pull/20729#issuecomment-1235029063
Closing in favor of https://github.com/apache/superset/pull/21066.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org