You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "giftig (via GitHub)" <gi...@apache.org> on 2023/10/31 12:36:32 UTC
[PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
giftig opened a new pull request, #25809:
URL: https://github.com/apache/superset/pull/25809
### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Reimplement, in a simpler way, part of the PRESTO_EXPAND_DATA feature for trino.
Make use of the trino library's type expansion logic added since the original feature was introduced, as we now have sqla ROW types parsed out recursively in a way we can analyse.
Analyse those ROWs and expand our definition of get_columns out to include dotted.path references to all fields it's possible to query by dotted path (i.e. all ROWs, nested arbitrarily deep).
Add an extra optional query_as field to the column definition so that we can override the way it's queried: otherwise sqlalchemy is going to quote the whole thing as a single column name, which isn't correct, it should actually be quoted per dotted segment, and we'll want to alias it to the full dotted string.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
<!--- Skip this if not applicable -->
:exclamation: I'll add these momentarily, please shout at me if I forget.
### TESTING INSTRUCTIONS
<!--- Required! What steps can be taken to manually verify the changes? -->
1. Query tables in trino containing rows and/or nested rows
2. Observe that dotted paths are expanded out on the schema preview, and queried in the table preview
To easily construct some data containing nested rows, you can use the following instructions:
1. Run `trinodb/trino:latest` in docker
2. `docker exec -it trino trino` to cli into it
3. Create a table with nested fields like this:
```sql
USE memory.default
CREATE TABLE foo AS (
WITH
data AS (
SELECT
(
SELECT
ARRAY[1, 2, 3] arr1,
ARRAY[4, 5, 6] arr2,
(
SELECT
(SELECT 'foo' foo, 'bar' bar) nest1a,
(SELECT 'hodor' foo, 'HODOR HODOR HODOR' bar) nest1b
) nest1,
(
SELECT
(SELECT 'baz' baz, 'buzz' buzz) nest2a,
(SELECT 'bzzzzzzt' buzzing, 'BZZZZZT!' buzzsaw) nest2b
) nest2
) field1,
'toplevel' field2
)
SELECT * FROM data
);
```
4. Connect to the dockerised trino via superset; I did this by simply adding a trino into the standard docker-compose definiton
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [x] Has associated discussion: https://github.com/apache/superset/discussions/25713#discussioncomment-7421689
- [x] Required feature flags: `TRINO_EXPAND_ROWS`
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec, which is unfortunate.
I can also just add some extra options to the base signature and pass the flag in directly rather than a reference to the database, but then the `Database` object is forced to care about a trino-specific option, which seems incorrect. If I phrase the config as something more generic like "format options" and take a dict then that might be ok, I'll give that a try.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377906965
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Rather than this being a feature flag (specific to Trino) it seems like this would make sense if this was a configuration at the database level. There's an `extra` column which has the ability to encode these types of configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1379807180
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
I see the options you've pointed to there are some of the checkboxes exposed on the form, yeah: https://github.com/apache/superset/blob/master/superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx#L192
I can update this form, I'm just wondering if there's any particular approval process for copy, and how the translations get filled out. Maybe I'm just making a mountain out of a molehill. I'll add some sensible label for now and leave it for review. ~I may even be able to show this option only when configuring a trino db, let me see if that's relatively simple.~ looks like we haven't attempted to do that with other engine-specific properties so I'll leave it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387990083
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Not sure, I just copied how the other checkboxes here were implemented; I'm not 100% clear on what the `indeterminate` functionality is for, though I notice they're all using `indeterminate={false}` here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1802664807
The backend changes LGTM. @michael-s-molina or @villebro would you mind taking a look at the frontend changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1379226435
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
@giftig there are [existing patterns](https://github.com/apache/superset/blob/a3686459a9ac4a4b916c111971de85800c0e7dee/superset/models/core.py#L216-L237) where we store configurations at the root level of the `dbs.extra` column. See [here](https://github.com/apache/superset/blob/a3686459a9ac4a4b916c111971de85800c0e7dee/superset/views/database/mixins.py#L123-L152) as well.
I speculate this information was exposed when the database connection editor modal was rewritten from the original Flask-AppBuilder view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377952643
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
@giftig I think the [metadata_params](https://github.com/apache/superset/blob/1e37f0b41782d35d68f657bfa87aeb9055d1e6e7/superset/models/core.py#L157) subfield within the `extra` column might be the best place to add this as it will allow this to be configurable via the UI. Otherwise one one would need to manage this via a SQL UPDATE statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1802310187
Hi @john-bodley @villebro @michael-s-molina gentle nudge about this one when you have some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378664925
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
I've made the change but I actually get a frontend error at the point of finishing writing a JSON object if I try to configure this in the UI:

which blanks out the page entirely as soon as I hit the closing brace. I'll raise a bug if we're not already tracking this. It exhibits very strange behaviour actually, e.g. if I type `{}` it wordlessly empties the box.
Edit: raised a bug: https://github.com/apache/superset/issues/25818
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1818446526
Hi @michael-s-molina, @justinpark have you had chance to test this one? Happy to merge this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387990083
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Not sure, I just copied how the other checkboxes here were implemented; I'm not 100% clear on what the `indeterminate` functionality is for, though I notice they're all using `indetermine={false}` here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377906965
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Rather than this being a feature flag (specific to Trino) I wonder if this would make sense if this was a configuration at the database level. There's an `extra` column which has the ability to encode these types of configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1379807180
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
I see the options you've pointed to there are some of the checkboxes exposed on the form, yeah: https://github.com/apache/superset/blob/master/superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx#L192
I can update this form, I'm just wondering if there's any particular approval process for copy, and how the translations get filled out. Maybe I'm just making a mountain out of a molehill. I'll add some sensible label for now and leave it for review. I may even be able to show this option only when configuring a trino db, let me see if that's relatively simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387995533
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Ah `Checkbox` also doesn't have an `id`, which the logic here needs to collect this data into a payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "justinpark (via GitHub)" <gi...@apache.org>.
justinpark commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1819529299
@giftig LGTM with the current UI design.
One thing I would like to revise the response structure to accommodate a future tree view user interface (UI) that is organized hierarchically. This UI will consist of columns, with each column containing child elements.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec and I'd have to touch half a dozen unrelated specs, which is unfortunate.
I can also just add an extra defaulted config parameter to the base signature and pass the flag in directly rather than a reference to the database, but then the `Database` object is forced to care about a trino-specific option, which seems incorrect. If I phrase the config as something more generic like "format options" and take a dict then that might be ok, I'll give that a try.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378677637
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
It doesn't look like there's anywhere else I can put this that can be edited from the UI without making form changes. I'll have a look and see if I can use a new top-level field within `extra` by updating the form, but I assume making form changes will necessitate frontend changes, new wording, translations, and seems like a whole can of worms.
So I'd propose we just go back to the feature flagged version for simplicity for the moment, but open to further suggestions.
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Ok so I can't put it into `metadata_params`; that's not just arbitrary database metadata, it's actually meant to represent the sqlalchemy `MetaData` class and is being extracted out like this:
```python
from sqlalchemy import MetaData
# ...
MetaData(**extra.get("metadata_params", {}))
```
so if I add something arbitrary to it (I had to do it directly in the db since the UI is broken) I just get:
```python
TypeError: __init__() got an unexpected keyword argument 'schema_options'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378666715
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Ok so I can't put it into `metadata_params`; that's not just arbitrary database metadata, it's actually meant to represent the sqlalchemy `MetaData` class and is being extracted out like this:
```python
from sqlalchemy import MetaData
MetaData(**extra.get("metadata_params", {}))
```
so if I add something arbitrary to it (I had to do it directly in the db since the UI is broken) I just get:
```python
TypeError: __init__() got an unexpected keyword argument 'schema_options'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378664925
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
I've made the change but I actually get a frontend error at the point of finishing writing a JSON object if I try to configure this in the UI:

which blanks out the page entirely as soon as I hit the closing brace. I'll raise a bug if we're not already tracking this. It exhibits very strange behaviour actually, e.g. if I type `{}` it wordlessly empties the box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378666715
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Ok so I can't put it into `metadata_params`; that's not just arbitrary database metadata, it's actually meant to represent the sqlalchemy `MetaData` class and is being extracted out like this:
```python
from sqlalchemy import MetaData
MetaData(**extra.get("metadata_params", {}))
```
so if I add something arbitrary to it I just get:
```python
TypeError: __init__() got an unexpected keyword argument 'schema_options'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec and I'd have to touch half a dozen unrelated specs, which is unfortunate.
I can also just add some extra optional kwargs to the base signature and pass the flag in directly rather than a reference to the database, but then the `Database` object is forced to care about a trino-specific option, which seems incorrect. If I phrase the config as something more generic like "format options" and take a dict then that might be ok, I'll give that a try.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec and I'd have to touch half a dozen unrelated specs, which is unfortunate.
I can also just add some extra options to the base signature and pass the flag in directly rather than a reference to the database, but then the `Database` object is forced to care about a trino-specific option, which seems incorrect. If I phrase the config as something more generic like "format options" and take a dict then that might be ok, I'll give that a try.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1803852722
@justinpark Should we test this PR with our database to see how the data is displayed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1388011037
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Thank you for the additional context. It looks like we need to revisit these components given the inconsistencies. `IndeterminateCheckbox` shouldn't have a `indeterminate` property. This is not part of this PR though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387969324
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Why are you using an indeterminate checkbox if you only have 2 possible states? Isn't the `Checkbox` component sufficient for your use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #25809:
URL: https://github.com/apache/superset/pull/25809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1379967266
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377952643
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
@giftig I think the [metadata_params](https://github.com/apache/superset/blob/1e37f0b41782d35d68f657bfa87aeb9055d1e6e7/superset/models/core.py#L157) subfield might be the best place to add this as it will allow this to be configurable via the UI. Otherwise one one would need to manage this via a SQL UPDATE statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377946799
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Makes sense, I didn't look into opening that can of worms as I feared it'd involve db migrations and form changes but if it's being encoded via `extra` it doesn't sound too hard, I'll take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1787708033
@john-bodley there are some difficulties in expanding out the data to be more readable which I thought about while working on this:
1. we can't really do much about arrays; the `PRESTO_EXPAND_DATA` thing tried to spread them across multiple rows in the output but this seemed very confusing for a few reasons (I went into some detail in the linked discussion)
2. the original way that was being done was too fragile; it was actually parsing the junk you see there and manually expanding it out, without awareness of types and with too much reliance on being able to parse the syntax we see here. I considered a part 2 to this work being to do that in a more sensible way, either wrapping the user's query with an exhaustive SELECT on the fields to expand out the ROWs for you, or giving the user more control by providing some macros to just make it easier to do exactly that.
There's more, I left a fair few notes on some challenges I found on the related discussion if you're interested.
But I think this fixes a good chunk of the problem and I have no prejudice on whether we also add on some display improvements as well; it's fairly well-encapsulated piece so hopefully it'll work fine with what @justinpark is working on.
Currently from our end we consider it acceptable that querying a `ROW` or querying `SELECT *` directly will produce an unreadable blob since users can see the field breakdown in the schema, see more readable data in the table preview, and autocomplete `foo.bar.baz` to let them query fields more directly. We'd rather users selected a handful of relevant fields than build datasets with hundreds of selected fields anyway. We did notice that when we query via the trino cli we actually get a slightly different and nicer format though, like this:
```
{foo=123, bar="something"}
```
which is a lot easier to understand than `[123, "something"]` with no labels. So we intended to try to get that same format into superset as our next step, but if there's already work being done to make it a tree we can expand out then sounds like a better solution is already in progress. I'll take a look at the SIP, we're definitely be interested in what's being done there and this stuff is very fresh in my mind right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378666715
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Ok so I can't put it into `metadata_params`; that's not just arbitrary database metadata, it's actually meant to represent [the sqlalchemy `MetaData` class](https://github.com/sqlalchemy/sqlalchemy/blob/main/lib/sqlalchemy/sql/schema.py#L5325) and is being extracted out like this:
```python
from sqlalchemy import MetaData
# ...
MetaData(**extra.get("metadata_params", {}))
```
so if I add something arbitrary to it (I had to do it directly in the db since the UI is broken) I just get:
```python
TypeError: __init__() got an unexpected keyword argument 'schema_options'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec and I'd have to touch half a dozen unrelated specs, which is unfortunate. I can't see a good way around this either; I'll need to change the base signature with whichever approach I use.
I'll go for minimally-invasive, at least, and use an optional, defaulted `options` argument with some arbitrary config to be understood by the specific backend. That at least saves me adding new arguments to all the test invocations of this method.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1378578983
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
There's a small wrinkle here in that the `get_columns` method doesn't have a reference to the specific database, it's just a classmethod on the db spec. I could change the signature to take one; it's invoked from `Database.get_columns` anyway so there'd definitely be one available there to pass in, but then this will change the signature of a core API on the base spec and I'd have to touch half a dozen unrelated specs, which is unfortunate. I can't see a good way around this either; I'll need to change the base signature with whichever approach I use.
I actually wonder why we don't an instance of the `EngineSpec` with the `Database` and some other context present, rather than using static class methods, but I assume there was a reason for that original choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1377906965
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
Rather than this being a feature flag (specific to Trino) it seems like this would make sense if this was a configuration at the database level. There's an `extra` column which has the ability to encode these types of configuration and/or the metadata parameters which is exposed in the database configuration modal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "justinpark (via GitHub)" <gi...@apache.org>.
justinpark commented on PR #25809:
URL: https://github.com/apache/superset/pull/25809#issuecomment-1819529436
@giftig LGTM with the current UI design.
One thing I would like to revise the response structure to accommodate a future tree view user interface (UI) that is organized hierarchically. This UI will consist of columns, with each column containing child elements.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1379807180
##########
superset/config.py:
##########
@@ -432,6 +432,11 @@ class D3Format(TypedDict, total=False):
# expanded into extra columns and/or arrays. This is experimental,
# and doesn't work with all nested types.
"PRESTO_EXPAND_DATA": False,
+ # When this feature is enabled, nested ROW types in Trino will be fully
+ # described in SQL Lab when retrieving schemas and performing previews.
+ # Unlike PRESTO_EXPAND_DATA this will not attempt to expand ARRAY or MAP
+ # types as these can't be queried with a dotted.path.syntax like ROWs
+ "TRINO_EXPAND_ROWS": False,
Review Comment:
I see the options you've pointed to there are some of the checkboxes exposed on the form, yeah: https://github.com/apache/superset/blob/master/superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx#L192
I can update this form, I'm just wondering if there's any particular approval process for copy, and how the translations get filled out. I'll add some sensible label for now and leave it for review. I may even be able to show this option only when configuring a trino db, let me see if that's relatively simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "john-bodley (via GitHub)" <gi...@apache.org>.
john-bodley commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1380444536
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387992488
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
I notice `Checkbox` is missing the `title` and `labelText` properties that `IndeterminateCheckbox` has, so I suspect it's just being used for the accompanying label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "giftig (via GitHub)" <gi...@apache.org>.
giftig commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1387992488
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
I notice `Checkbox` is missing the `title` and `labelText` properties that `IndeterminateCheckbox` has, so I suspect it's just being used for the label + tooltip here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] feat(sqllab): TRINO_EXPAND_ROWS: expand columns from ROWs [superset]
Posted by "michael-s-molina (via GitHub)" <gi...@apache.org>.
michael-s-molina commented on code in PR #25809:
URL: https://github.com/apache/superset/pull/25809#discussion_r1388011037
##########
superset-frontend/src/features/databases/DatabaseModal/ExtraOptions.tsx:
##########
@@ -220,6 +220,22 @@ const ExtraOptions = ({
/>
</div>
</StyledInputContainer>
+ <StyledInputContainer>
+ <div className="input-container">
+ <IndeterminateCheckbox
Review Comment:
Thank you for the additional context. It looks like we need to revisit these components given the inconsistencies. `IndeterminateCheckbox` shouldn't have a `indeterminate` property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org