You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "LittleWat (via GitHub)" <gi...@apache.org> on 2023/11/29 13:47:18 UTC

[PR] fix(trino): the error when fetching Hudi tables Schema [superset]

LittleWat opened a new pull request, #26131:
URL: https://github.com/apache/superset/pull/26131

   <!---
   Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/
   Example:
   fix(dashboard): load charts correctly
   -->
   
   ### SUMMARY
   <!--- Describe the change below, including rationale and design decisions -->
   - This PR attempts to close #21945
   
   `$partitions` is not necessary so let me remove that.
   
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   <!--- Skip this if not applicable -->
   
   ### TESTING INSTRUCTIONS
   <!--- Required! What steps can be taken to manually verify the changes? -->
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [x] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #26131:
URL: https://github.com/apache/superset/pull/26131#issuecomment-1832440061

   Thanks @LittleWat for the PR. I've kicked off CI and added a few comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "LittleWat (via GitHub)" <gi...@apache.org>.
LittleWat commented on PR #26131:
URL: https://github.com/apache/superset/pull/26131#issuecomment-1868684642

   Let me close this PR because this was fixed by updating the trino-python-client (https://github.com/trinodb/trino-python-client/pull/426)
   
   ```
   pip install git+https://github.com/trinodb/trino-python-client.git@9df2cf2756d9bb189de719a80623064acfe65dea 
   ```
   
   Fixed the error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #26131:
URL: https://github.com/apache/superset/pull/26131#discussion_r1409676842


##########
superset/db_engine_specs/presto.py:
##########
@@ -467,7 +468,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
             full_table_name = f"{schema}.{table_name}" if schema else table_name
             partition_select_clause = f"SHOW PARTITIONS FROM {full_table_name}"
         else:
-            system_table_name = f'"{table_name}$partitions"'
+            system_table_name = f'"{table_name}$partitions"' if not is_trino else table_name

Review Comment:
   @LittleWat note that Presto does support the `$partitions` table suffix per the [release notes](https://prestodb.io/docs/current/release/release-0.199.html). Are you saying that Hudi (which I'm not well versed with) doesn't?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "LittleWat (via GitHub)" <gi...@apache.org>.
LittleWat closed pull request #26131: fix(trino): the error when fetching Hudi tables Schema
URL: https://github.com/apache/superset/pull/26131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #26131:
URL: https://github.com/apache/superset/pull/26131#discussion_r1409678137


##########
superset/db_engine_specs/presto.py:
##########
@@ -427,6 +427,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
         limit: int = 0,
         order_by: list[tuple[str, bool]] | None = None,
         filters: dict[Any, Any] | None = None,
+        is_trino: bool = False,

Review Comment:
   If this is the route we want to take, ideally this should be not Trino specific, i.e., maybe something like `use_partition_suffix`. 



##########
superset/db_engine_specs/presto.py:
##########
@@ -467,7 +468,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
             full_table_name = f"{schema}.{table_name}" if schema else table_name
             partition_select_clause = f"SHOW PARTITIONS FROM {full_table_name}"
         else:
-            system_table_name = f'"{table_name}$partitions"'
+            system_table_name = f'"{table_name}$partitions"' if not is_trino else table_name

Review Comment:
   @LittleWat note that Presto does support the `$partitions` table suffix per the [release notes](https://prestodb.io/docs/current/release/release-0.199.html).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "LittleWat (via GitHub)" <gi...@apache.org>.
LittleWat commented on code in PR #26131:
URL: https://github.com/apache/superset/pull/26131#discussion_r1410167472


##########
superset/db_engine_specs/presto.py:
##########
@@ -467,7 +468,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
             full_table_name = f"{schema}.{table_name}" if schema else table_name
             partition_select_clause = f"SHOW PARTITIONS FROM {full_table_name}"
         else:
-            system_table_name = f'"{table_name}$partitions"'
+            system_table_name = f'"{table_name}$partitions"' if not is_trino else table_name

Review Comment:
   thank you for checking! Presto+Hudi works but Trino+Hudi does not work. I'm wondering if Trino supports the $partitions table suffix 🤔 



##########
superset/db_engine_specs/presto.py:
##########
@@ -467,7 +468,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
             full_table_name = f"{schema}.{table_name}" if schema else table_name
             partition_select_clause = f"SHOW PARTITIONS FROM {full_table_name}"
         else:
-            system_table_name = f'"{table_name}$partitions"'
+            system_table_name = f'"{table_name}$partitions"' if not is_trino else table_name

Review Comment:
   thank you for checking! Presto+Hudi works but Trino+Hudi does not work. I'm wondering if Trino supports the $partitions table suffix 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26131:
URL: https://github.com/apache/superset/pull/26131#issuecomment-1832459178

   ## [Codecov](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: `3 lines` in your changes are missing coverage. Please review.
   > Comparison is base [(`b9aaf0f`)](https://app.codecov.io/gh/apache/superset/commit/b9aaf0fc057d44e99bcdb1c63b362b5608781644?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 58.93% compared to head [(`1abbfe8`)](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 66.33%.
   > Report is 3 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [superset/db\_engine\_specs/trino.py](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGJfZW5naW5lX3NwZWNzL3RyaW5vLnB5) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #26131      +/-   ##
   ==========================================
   + Coverage   58.93%   66.33%   +7.40%     
   ==========================================
     Files        1941     1941              
     Lines       75873    75849      -24     
     Branches     8447     8447              
   ==========================================
   + Hits        44714    50317    +5603     
   + Misses      28984    23357    -5627     
     Partials     2175     2175              
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hive](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.68% <0.00%> (-0.01%)` | :arrow_down: |
   | [presto](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.63% <0.00%> (-0.01%)` | :arrow_down: |
   | [python](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.20% <25.00%> (+15.41%)` | :arrow_up: |
   | [sqlite](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.90% <25.00%> (?)` | |
   | [unit](https://app.codecov.io/gh/apache/superset/pull/26131/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/26131?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


Re: [PR] fix(trino): the error when fetching Hudi tables Schema [superset]

Posted by "LittleWat (via GitHub)" <gi...@apache.org>.
LittleWat commented on code in PR #26131:
URL: https://github.com/apache/superset/pull/26131#discussion_r1410208494


##########
superset/db_engine_specs/presto.py:
##########
@@ -467,7 +468,7 @@ def _partition_query(  # pylint: disable=too-many-arguments,too-many-locals,unus
             full_table_name = f"{schema}.{table_name}" if schema else table_name
             partition_select_clause = f"SHOW PARTITIONS FROM {full_table_name}"
         else:
-            system_table_name = f'"{table_name}$partitions"'
+            system_table_name = f'"{table_name}$partitions"' if not is_trino else table_name

Review Comment:
   even after I made this change, I got the error so I made a draft PR in trino-python-client repo:
   https://github.com/trinodb/trino-python-client/pull/426



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org